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“think  piece”  on  how  to  understand  and  address  the  unique  analytical  challenges 
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proliferation,  and  organized  crime.  Co-authored  by  Warren  Fishbein 
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RAND  Corporation,  the  paper  proposes  some  practical  ideas  for  adapting  the 
organizational  culture  and  processes  in  which  analysis  of  these  issues  is  done 
to  improve  understanding  and  warning. 
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Transnational  Issues.”  (Reports  of  these  workshops  are  published  separately  by 
RAND  Corporation  in  report  CF-200.)  In  this  paper,  workshop  insights  are  coupled 
with  findings  from  further  research  on  concepts  such  as  intuitive  thinking,  sense¬ 
making,  and  mindfulness  to  suggest  an  approach  for  applying  what  the  authors  call 
“alternative  sense-making”  to  complex  transnational  issues. 

The  ideas  suggested  here,  however,  are  less  a  prescription  for  analytical  practice 
than  an  invitation  to  dialogue,  debate,  and  further  research  that  will  help  advance 
the  doctrine  of  analysis  for  transnational  threats.  The  Kent  Center  welcomes  this 
contribution  to  the  literature  on  intelligence  analysis  and  looks  forward 
to  continued  exploration  of  the  arguments  presented  here  and  in  an  abridged 
version,  “Rethinking  ‘Alternative  Analysis’  to  Address  Transnational  Threats,” 
published  in  Kent  Center  Occasional  Papers,  Volume  3,  Number  2. 
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Summary 


Intelligence  Community  analytic  organizations  need  to  institutionalize  processes 
to  consider  whether  and  how  they  might  “have  gotten  it  wrong”  to  enhance  their 
abilities  to  anticipate  potential  threats  in  highly  complex,  fast-moving  transnational 
issues,  such  as  terrorism  and  weapons  proliferation.  Such  processes  would 
involve  sustained,  collaborative  efforts  by  analysts  to  question  their  judgments  and 
underlying  assumptions,  employing  both  critical  and  creative  modes  of  thought.  For 
such  processes  to  be  effective,  significant  changes  in  the  cultures  and  business 
processes  of  analytic  organizations  will  be  required. 

These  are  the  key  conclusions  arising  from  a  project  undertaken  by  the  Kent 
School’s  Global  Futures  Partnership  and  the  RAND  Corporation  to  rethink 
“alternative  analysis” — tools  designed  to  help  analysts  and  decisionmakers  employ 
rigorous  self-review,  question  judgments,  and  explore  alternative  outcomes — to 
better  address  threats  in  the  increasingly  important  realm  of  transnational  issues. 

In  a  series  of  unclassified  workshops,  Intelligence  Community  analysts  and  analytic 
managers  came  together  on  a  nonattribution  basis  with  outside  thinkers  in  a  broad 
range  of  fields  relevant  to  the  analytic  process,  including  cognitive  psychology, 
psychiatry,  organizational  behavior,  artificial  intelligence,  knowledge  management, 
intelligence  studies,  and  the  foreign  policy  process.  Through  presentations  and 
discussions  among  participants,  the  workshops  sought  to  generate  broad  concepts 
about  adapting  alternative  analysis  to  enhance  warning  of  out-of-the-ordinary 
actions  undertaken  by  nonstate  actors,  epitomized  in  the  September  1 1  attacks. 
What  follows  in  this  report  are  some  of  the  more  intriguing  ideas  that  surfaced  at 
the  workshops,  arrayed  and  developed  by  the  project  leaders  into  a  systematic 
argument. 

If  traditional  intelligence  analysis  generates  forecasts  or  explanations  based 
on  logical  processing  of  available  evidence,  alternative  analysis  seeks  to  help 
analysts  and  policymakers  stretch  their  thinking  through  structured  techniques 
that  challenge  ongoing  assumptions  and  broaden  the  range  of  possible  outcomes 
considered.  Properly  applied,  it  serves  as  a  hedge  against  the  natural  tendencies 
of  analysts — like  all  human  beings — to  perceive  information  selectively  through  the 
lens  of  preconceptions,  to  search  too  narrowly  for  facts  that  would  confirm  rather 
than  discredit  existing  hypotheses,  and  to  be  unduly  influenced  by  premature 
consensus  within  analytic  groups  close  at  hand.  In  theory,  use  of  alternative 
analysis  techniques  can  help  to  reduce  the  likelihood  of  “intelligence  failures,”  which 
historically  have  stemmed  in  part  from  such  mental  errors  (e.g.  the  ingrained  belief 
that  the  Japanese  could  not  mount  a  successful  attack  against  Pearl  Harbor).  In 
reality,  however,  alternative  analysis  has  not  been  particularly  effective  within  the 
Intelligence  Community.  It  has  been  employed  only  sporadically  at  best,  and  more 
often  than  not  as  a  “nice-to-have”  supplement  tacked  on  to  traditional  analysis  rather 
than  integrated  at  the  outset  as  an  essential  component  of  the  analytic  enterprise  in 
a  world  of  uncertainty  and  deception. 

But  an  even  more  significant  problem  with  alternative  analysis,  looking  ahead,  is 
that  it  may  be  less  effective  at  enhancing  the  warning  process  for  threats  in  the 
transnational  realm  than  it  could  be,  at  least  theoretically,  in  the  more  traditional 
state-to-state  realm.  This  is  so  because  alternative  analysis,  like  analysis  of  any 
kind,  involves  the  application  of  structured  argumentation  to  address  discrete 
questions — what  if  Ruritania  acquires  “the  Bomb?”Yet,  many  issues  in  the 


transnational  realm  may  not  lend  themselves  as  readily  to  such  formal  approaches 
as  do  more  traditional  types  of  intelligence  problems. 

Analysis  involves  breaking  down  problems  into  constituent  parts,  such  as  causes  and 
effects,  and  using  logical  operations  to  identify  and  test  hypotheses  for  the  purposes 
of  explanation  and  prediction.  Analysis  is  relevant  for  bounded  problems  in  which 
there  is  a  restricted  range  of  outcomes  or  hypotheses  to  be  evaluated.  Many  classic 
state-to-state  intelligence  issues  in  which  there  are  well-established  patterns  fit  into 
this  category,  including  the  status  of  a  rival’s  military  capabilities  or  foreign  policy 
objectives.  These  issues  are  either,  to  use  familiar  terminology  in  intelligence  studies, 
“puzzles”  (theoretically  soluble  if  denied  information  were  obtained)  or  “mysteries” 
(which  are  future  and  contingent  and  thus  not  soluble,  but  for  which  several  possible 
outcomes  can  be  identified  and  ranked  in  likelihood).  Traditional  analysis  can  be 
effectively  deployed  against  bounded  problems,  and  alternative  analysis  tools  can 
ensure  that  more  than  one  conclusion  or  outcome  is  seriously  considered. 

However,  intelligence  problems  in  the  transnational  realm  are,  generally  speaking, 
less  bounded  than  are  those  in  the  state-to-state  realm.  Transnational  groups,  like 
terrorist  cells  or  proliferation  networks,  are  likely  to  be  smaller  and  more  numerous 
than  states,  less  constrained  by  rules  and  historic  precedent,  and  more  affected  by 
tactical  and  situational  circumstances,  as  opposed  to  deeper  internal  drivers.  They 
are  thus  more  difficult  to  understand  and  more  capable  of  engaging  in  a  wide  array  of 
unprecedented  behaviors. 

To  be  sure,  the  distinction  between  transnational  and  traditional  intelligence 
problems  should  not  be  overstated:  there  are  some  state-to-state  problems,  such 
as  battlefield  situations  or  crisis  diplomacy,  where  situationally  driven  interactions 
among  a  large  number  of  players  also  can  produce  a  wide  variety  of  outcomes. 

For  either  transnational  or  traditional  intelligence  problems  that  are  subject  to  high 
levels  of  uncertainty — a  third  category  that  can  be  called  “complexities” — analysis 
is  unlikely  to  prove  very  effective  beyond  a  very  general  level.  There  are  simply  too 
many  possible  causes  and  effects  and  too  little  reliable  data  to  be  able  to  disprove 
hypotheses. 

Such  issues  can  be  comprehended,  however,  through  a  process  of  “sense-making.” 
This  concept,  developed  by  management  scientists  to  help  business  and  public- 
sector  organizations  cope  with  uncertainty,  also  has  clear  applications  to  intelligence. 
Sense-making  is  a  continuous,  iterative,  largely  intuitive  effort  to  paint  a  picture 
of  what  is  going  on  in  the  environment  of  an  organization.  It  is  accomplished  by 
comparing  new  events  to  past  patterns,  or  in  the  case  of  anomalies,  by  developing 
stories  to  account  for  them.  A  process  somewhat  akin  to  sense-making  is  done 
everyday  in  the  Intelligence  Community  in  current  intelligence,  which  is  a  continuous, 
largely  informal  effort  to  update  the  story  line  on  an  issue.  It  also  underlies  the 
key  warning  concept  of  “connecting  the  dots,”  which  implies  holistic  recognition  or 
discovery  of  patterns  of  behavior. 

Reducing  the  impact  of  cognitive  or  organizational  biases  on  judgment  in  the 
sense-making  process  requires  an  approach  different  from  that  of  alternative 
analysis.  The  aim  would  not  be  to  rigorously  examine  alternative  assumptions  or 
outcomes  but  rather  to  prompt  analysts  to  be  continually  on  the  lookout  for  different 
types  of  patterns.  It  would  be,  to  employ  another  concept  used  by  organizational 
decisionmaking  experts,  to  promote  “mindfulness”  within  the  analytic  intelligence 


organization.  This  intellectual  orientation,  favoring  continuous  evaluation  of 
expectations  and  assumptions,  is  found  in  many  organizations  that  successfully 
deal  with  high  levels  of  complexity  and  uncertainty-such  as  aircraft  carriers  and 
nuclear  power  plants.  Such  organizations  do  very  effective  sense-making  of  their 
environments,  as  is  indicated  by  exceptionally  low  rates  of  accident  (their  version  of 
“intelligence  failure”).  High  levels  of  mindfulness  are  associated  with,  among  other 
things,  a  preoccupation  with  failure,  both  past  and  potential,  and  a  “learning  culture” 
in  which  it  is  safe  and  even  valued  for  members  of  the  organization  to  admit  error 
and  raise  doubts. 

For  intelligence,  enhancing  mindfulness  would  be  a  process,  not  a  tool.  That  process 
would  be: 

•  Continual,  not  discrete  or  “one-off”  efforts.  The  objective  would  be  to 
regularly  explore  different  possible  outcomes  and  debate  assumptions, 
linked  to  all  incoming  information  that  could  possibly  relate  to  the  issue  under 
consideration. 

•  Creative,  freewheeling,  in  place  of  the  more  formal  alternative  analytic 
process  with  its  strong  emphasis  on  logical  argument  to  come  to  clear 
conclusions.  It  would  consciously  mix  mental  biases-for  instance,  building 
teams,  akin  to  the  practice  of  some  Wall  Street  firms,  known  as  “barbelling,” 
which  involves  pairing  young  financial  professionals  with  those  over  50  to  take 
advantage  of  both  adventurousness  and  caution  born  of  experience.  And  it 
would  provide  increased  time  for  reflection  and  conversation,  for  novel  ideas 
often  “pop  out”  of  slow-moving,  largely  unconscious,  contemplative  modes  of 
thought,  rather  than  out  of  more  conscious,  purposeful,  and  analytic  modes. 

•  Collaborative,  because  transnational  issues  inevitably  involve  teams  or 
even  networks  of  analysts  given  that  these  issues  cross  both  national  and 
functional  boundaries.  Indeed,  sense-making  might  be  “out  loud” — orally 
reviewing  assumptions  and  alternative  outcomes  in  a  collective  forum. 

•  Counterintuitive,  seeking  disconfirming  evidence,  not  confirming  evidence, 
for  instance,  by  featuring  regular — if  brief  and  informal — exercises  in  which 
analysts  focus  on  how  they  could  be  wrong. 

•  Consumer-friendly,  an  enormous  challenge  since  sustaining  “mindfulness” 
among  time-pressed  consumers  is  even  more  difficult  than  getting  them  to 
read  alternative  analysis  papers  on  occasion.  It  requires  new  and  engaging, 
experiential  intelligence  “products”  such  as  RapiSims — sophisticated 
spreadsheet-based  programs  that  allow  consumers  to  manipulate  variables 
to  explore  alternative  outcomes — and  enhanced  relationships  between 
analysts  and  consumers  to  facilitate  informal  dialogue  about  assumptions  and 
outcomes. 


Some  ways  in  which  these  processes  could  be  implemented  are  summarized  in  the 
table  below. 


Key  Practical  Ideas 


Idea 

Implementation  and  Purpose 

Develop  information  technology  to  store  and 
automatically  recover  hypotheses,  ideas. 

Aid  analysts’  memory  and  creative 
thinking,  and  promote  collaboration 

Employ  analytic  methodologists  with  training 
in  creativity  and  facilitation. 

Design  and  facilitate  divergent  thinking 
exercises  and  structured  dialogues 
aimed  at  surfacing  alternative  views. 

Consciously  mix  biases  in  teams  (e.g. 
“barbelling”). 

Increase  likelihood  of  alternative 
interpretations  of  evidence. 

Introduce  “out  loud  collaborative  sense¬ 
making”  processes,  both  in  person  and 
virtual. 

Structured  dialogues  to  consider  all 
possibilities. 

Use  web-logs  as  a  production  vehicle. 

Common,  continuous  platform  for 
carrying  out  a  “virtual  dialogue”  on 
alternatives. 

Regularly  do  after-action  reports. 

Look  at  failures  and  successes  with  an 
eye  to  drawing  constructive  lessons. 

Provide  opportunities  for  experiential 
learning  by  intelligence  consumers. 

Brief  simulations/games  to  help 
consumers  comprehend  range  of 
uncertainty. 

Promote  analyst  reflection  and  introspection. 

Allow  time  off-line  for  premortems  and 
after-action  exercises. 

In  sum,  alternative  analysis-or,  more  accurately,  alternative  “sense-making”-needs 
to  be  conceived  of  as  ongoing  organizational  processes  aimed  at  promoting 
sustained  mindfulness,  rather  than  as  a  set  of  tools  that  analysts  are  encouraged 
to  employ.  These  processes  would  require  creating  an  organizational  culture 
that,  through  exhortation,  training,  and  example,  values  continuous,  collective 
introspection — often  difficult  to  achieve  in  the  existing  high-demand,  understaffed 
situations  in  which  analysts  find  themselves.  Implementing  such  a  cultural  shift 
would,  therefore,  require  that  it  be  a  high  priority  of  senior  intelligence  managers, 
reinforced  by  changes  in  reward  structures,  production  schedules  and  staffing 
requirements. 

Could  mindfulness-focused  organizational  processes  really  enhance  warning 
of  emerging  transnational  threats?  No  one  can  confidently  answer  yes,  but 
reflecting  on  past  surprises  in  “complex”  situations  suggests  that  even  modest 
improvements  could  make  a  significant  difference  in  preparedness.  We  could  posit, 
for  example,  that  if  the  concerns  of  the  Phoenix  FBI  office  about  flight  training  before 
September  11  had  been  shared  broadly  within  the  government  and  integrated 
into  a  mindfulness-focused  inter-agency  process — featuring  collaborative  sense¬ 
making,  web-log  type  forums,  computer-generated  references  to  extant  scenarios 
for  crashing  airplanes  into  prominent  targets — they  might  have  garnered  far  broader 
attention  than  they  did. 


Introduction 

Heading  off  surprise  has  driven  US 
intelligence  since  the  attack  on  Pearl 
Harbor,  which  itself  was  responsible  for 
the  basic  shape  of  the  modern  American 
Intelligence  Community.  September  11 
was  a  powerful  impetus  to  the  debate  on 
how  to  do  better;  intelligence  produced 
shriller  and  shriller  strategic  warning 
of  terrorism  over  the  summer  of  2001 
but  could  not  follow  the  specific  trail 
of  hijackers  in  and  out  of  the  country, 
nor  provide  tactical  warning  of  specific 
threats  at  home. 

Most  of  the  criticism  after  September 
1 1  focused  on  the  ragged  sharing  of 
information  between  intelligence  and 
law  enforcement,  especially  the  CIA  and 
the  FBI,  and  the  way  the  FBI  conceived 
its  business.  Yet  the  attacks  also 
directed  attention  to  how  intelligence 
analysis  is  done,  as  was  reflected  in  the 
frequent  refrain  about  the  Intelligence 
Community’s  failure  to  “connect  the 
dots.”1  In  particular,  analysts  were 
alleged  to  have  suffered  some  version 
of  what  foreign  affairs  columnist 
Thomas  Friedman  called  a  “failure  of 
imagination.”2  That  is,  they  were  so 
fixed  on  the  risk  of  bombs  on  planes 
that  the  possibility  that  terrorists  might 
use  a  passenger  airplane  as  a  bomb 
had  receded  from  view.  More  generally, 
the  possibility  of  a  devastating  attack 
taking  place  on  American  soil,  while 
acknowledged,  was  not  fully  grasped. 


For  some,  like  Friedman  himself,  this 
was  an  almost  excusable  outcome  of 
the  collective  American  inability  to  grasp 
the  nature  of  “evil.”  For  others,  this  was 
a  less  pardonable  failure  to  forecast  a 
“predictable  surprise" — predictable  in 
that  there  had  been  previous  attempts 
or  plots  to  commandeer  aircraft  for  use 
as  weapons  that,  at  least  in  hindsight, 
should  clearly  have  shaped  analysis.3 

Over  the  years,  the  Intelligence 
Community,  as  well  as  other  public 
and  private-sector  organizations 
that  have  to  cope  with  uncertain 
futures,  have  developed  tools  for 
rigorous  self-review  to  help  avert 
“failures  of  imagination.”  These  tools 
have  collectively  become  known  as 
“alternative  analysis”  to  distinguish 
them  from  more  conventional  forms 
of  analysis  that  generate  forecasts  or 
explanations  based  on  logical,  typically 
linear  processing  of  available  evidence. 
Alternative  analysis  techniques  seek  to 
help  analysts  and  policymakers  stretch 
their  thinking  by  broadening  the  array 
of  outcomes  considered — examining 
“alternatives” — or  by  challenging 
underlying  assumptions  that  may 
constrain  thinking.  Alternative  analysis 
is  designed  to  hedge  against  the  natural 
tendency  of  analysts — like  all  human 
beings — to  focus  on  information  that 
confirms  rather  than  discredits  existing 
hypotheses,  or  to  be  unduly  influenced 
by  premature  consensus  within  analytic 
groups  close  at  hand.  Although  many 


The  findings  of  the  joint  House-Senate  investigation  of  September  1 1  outlines  the  basic  story.  It  is  Final 
Report,  Part  I,  The  Joint  Inquiry,  The  Context,  Part  I,  Findings  and  Conclusions,  1 0  December  2002.  A  fuller 
account  is  contained  in  Senator  Richard  Shelby’s  long  supplementary  document,  September  1 1  and  the 
Imperative  of  Reform  in  the  Intelligence  Community,  Additional  Views,  1 0  December  2002.  Both  are  available 
at  www.fas.ora/irD/conaress/2002  rpt/index.html  (last  visited  5  December  2003).  See,  in  particular,  Shelby’s 
report,  p.  15ff.  The  National  Commission  on  Terrorist  Attacks  Upon  the  United  States  also  lays  out  the  story, 
and  it  adds  some  fresh  details.  See  its  report,  The  9/1 1  Commission  Report,  (Washington,  2004),  available  at 
http://www.9-1 1  commission.gov/  (last  visited  28  July  2004). 

p 

Thomas  Friedman,  “A  Failure  to  Imagine,  “  New  York  Times,  1 4  May  2002. 

3  It  is  not  as  though  planes  as  bombs  had  not  been  considered.  The  concern  runs  back  at  least  as  far  as  the 
late  1 980s.  See,  for  instance,  Brian  M.  Jenkins,  The  Terrorist  Threat  to  Commercial  Aviation,  P-7450,  (Santa 
Monica:  RAND,  March  1989),  p.  10:  “The  nightmare  of  governments  is  that  suicidal  terrorists  will  hijack  a 
commercial  airliner  and,  by  killing  or  replacing  its  crew,  crash  into  a  city  or  some  vital  facility.”  Rather,  the 
concern  receded  from  view  as  attention  shifted  to  other  kinds  of  threats. 
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The  starting 
assumption  was 
that  transnational 
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a  different  set 
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intelligence 
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of  the  techniques  of  alternative  analysis 
have  been  around  in  some  form  for 
many  years,  only  in  the  1 990s  did  the 
Intelligence  Community  deem  these 
tools  an  essential  component  of  the 
overall  analytic  effort. 

In  2003  the  Global  Futures  Partnership 
(GFP)  in  the  Directorate  of  Intelligence’s 
Sherman  Kent  School  for  Intelligence 
Analysis  and  the  RAND  Corporation 
embarked  upon  a  project  to  reconsider 
alternative  analysis  in  light  of  the 
growing  importance  of  transnational 
issues,  primarily  terrorism,  but  also 
including  organized  crime  and  weapons 
proliferation,  among  others.  The  starting 
assumption  was  that  transnational 
issues  presented  a  different  set  of 
analytic  challenges  than  did  more 
traditional  intelligence  topics  targeted 
primarily  on  nation  states. 


The  project  focused  more  on  how  to 
effectively  integrate  alternative  analysis 
into  the  overall  analytic  and  policymaking 
process  for  transnational  issues  than  on 
evaluating  specific  tools  or  developing 
new  ones.  This  emphasis  reflected 
the  view  that  whether  for  traditional  or 
transnational  issues,  alternative  analysis 
typically  is: 

•  Used  only  episodically  in  the 
analytic  process; 

•  Often  viewed,  at  best,  as  a 

“nice  to  have  “  supplement  tacked 
on  to  conventional  analysis 


rather  than  as  an  essential 
component  of  the  overall  analytic 
process; and 

•  Not  particularly  effective  in 
influencing  the  policy  process. 

The  classic  example  of  Pearl  Harbor 
provides  vivid  testimony  in  support  of  the 
last  point.  As  CIA  University  President 
Frans  Bax  has  pointed  out,  senior  US 
Navy  officers  produced  an  alternative 
analysis-like  secret  report  in  early  1941 
that  quite  presciently  explored  the  then 
somewhat  outlandish  possibility  of  an 
aerial  torpedo  attack  launched  from 
aircraft  carriers.* * 4  The  report  was  initially 
read  with  interest  by  senior  officials  but 
eventually  ended  up  gathering  dust  on 
a  classified  bookshelf.  It  did  not  appear 
to  have  influenced  thinking  in  the 
immediate  runup  to  December  7th  when 
even  modest  increases  in  readiness  or 
alertness  could  have  made  a  significant 
difference.5  It  is  with  the  goal  of  finding 
ways  to  keep  alternative  analysis  from 
similarly  “gathering  dust”  in  the  present 
era — so  rife  with  potential  discontinuity 
that  it  has  already  been  dubbed  the 
“age  of  surprise” — that  this  project  was 
undertaken.6 7 

To  this  end,  GFP  and  RAND  convened 
a  series  of  unclassified  one-day 
workshops  on  better  integrating 
alternative  analysis  into  the  analytic 
process.'  It  examined  that  question 
from  several  different  viewpoints — that 
of  the  individual  analyst,  the  intelligence 


Frans  R.  Bax,  “Intelligence  Lessons  From  Pearl  Harbor,”  Studies  in  Intelligence,  November  2002,  pps.  1-9. 

The  classic  work  on  the  intelligence  failure  at  Pearl  Harbor  is  Roberta  Wohlstetter,  Pearl  Harbor.  Warning  and 

Decision,  (Stanford:  Stanford  University  Press,  1962). 

5  Eliot  Cohen  and  John  Gooch,  Military  Misfortune'.  The  Anatomy  of  Failure  in  War,  (New  York:  Vintage,  1991), 
pp.  50-51 

Elliot  A.  Cohen,  “A  Tale  of  Two  Secretaries,”  Foreign  Affairs,  May/June  2002,  pp.33-46. 

7  Reports  of  the  not-for-attribution  workshops  are  published  in  a  separate  RAND  report,  CF-200.  As  might  be 
expected  from  a  series  of  gatherings  of  distinguished  and  diverse  minds,  discussions  dealt  with  the  issue  at 
hand  but  also  ranged  well  beyond  to  include  general  issues  of  improving  the  intelligence  process.  The  individual 
workshop  reports  are  thus  well  worth  reading. 


2 


organization,  and  the  policymaker. 

The  workshops  brought  together 
analysts  from  the  CIA’s  Directorate  of 
Intelligence  and  from  other  agencies 
focused  on  transnational  issues, 
along  with  a  distinguished  group  of 
over  30  nongovernmental  experts. 

These  experts  came  from  a  variety 
of  disciplines  relevant  to  thinking 
about  the  analytic  process:  cognitive 
psychology,  psychiatry,  group  dynamics, 
information  technology,  organizational 
studies,  knowledge  management, 
artificial  intelligence,  diplomatic  history, 
technology  studies,  strategic  studies, 
and  journalism,  along  with  experts  in 
specific  transnational  domains  such  as 
terrorism  and  proliferation.  The  aim  of  the 
workshops — which  featured  both  formal 
presentations  and  break-out  group 
discussions — was  to  blend  the  widely 
varied  perspectives  of  the  participants 
with  the  aim  of  generating  new  ideas 
that  could  ultimately  yield  more  concrete 
proposals.  What  follows  here  is  a 
synthesis  of  key  findings  about  applying 
alternative  analysis  to  transnational 
issues,  coupled  with  the  results  of  further 
research  undertaken  by  the  project 
leaders  in  response  to  ideas  raised  in  the 
workshops. 

The  Logic  of  Alternative  Analysis 

The  underlying  rationale  for  the 
incorporation  of  alternative  analysis 
techniques  into  the  analytic  process 
has  to  do  both  with  the  complexity  of 
the  subject  matter  of  intelligence  and 
with  the  limitations  of  the  human  mind. 
Intelligence  addresses,  almost  by 
definition,  issues  characterized  by  high 
uncertainty,  such  as  the  intentions  of 
foreign  actors,  their  capabilities  when 


they  are  determined  to  keep  those 
hidden,  and  the  outcomes  of  complex, 
interactive  systems  such  as  economies 
and  the  international  system  as  a 
whole.8 

The  information  base  available  to 
analyze  these  and  other  types  of 
issues — both  openly  available  and 
clandestinely  acquired — is  huge, 
and  it  is  a  great  challenge  to  pick  out 
diagnostic  information  from  the  torrent 
of  irrelevant  data  (the  “signals”  from 
the  “noise”).  Moreover,  unlike  other 
intellectual  endeavors,  intelligence 
analysis  often  confronts  efforts  to 
shroud  key  information  in  secrecy  or  to 
actively  mislead  by  planting  deceptive 
information.  For  any  intelligence  issue, 
prediction  is  at  least  very  difficult, 
and  thus  it  is  useful  to  systematically 
consider  alternative  explanations  and 
outcomes. 

But  perhaps  an  even  more  important 
reason  to  consider  alternatives  involves 
the  fact  that  analysts  do  not  process 
information  entirely  “objectively”  because 
of  biases  in  human  perception  and 
judgment.  As  Richards  Heuer  and  others 
have  argued,  all  individuals  assimilate 
and  evaluate  information  through  the 
medium  of  “mental  models”  (sometimes 
also  called  “frames”  or  “mind-sets”).9 
These  are  experience-based  constructs 
of  assumptions  and  expectations  both 
about  the  world  in  general  and  more 
specific  domains.  These  constructs 
strongly  influence  what  we  “take  on 
board,”  with  information  that  it  is  in 
accordance  with  our  models  more  likely 
to  be  perceived  and  remembered  than 
information  that  is  at  variance  with 
them.  They  are  all  the  more  powerful 


8  Challenges  posed  to  analysis  by  uncertainty  are  examined  in  greater  depth  later  because  there  is  a  case  to 
be  made  (or  at  least  further  explored)  that  transnational  issues  involve  a  significantly  higher  level  of  uncertainty 
than  more  traditional  intelligence  topics  and  thus  requires  a  different  approach  to  alternative  analysis. 

g 

Richards  S.  Heuer,  Jr.,  The  Psychology  of  Intelligence  Analysis,  (Washington:  Center  for  the  Study  of 
Intelligence,  1999). 
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when  analysts  or  leaders  are  fatigued 
and  thus  lose  the  ability  to  think  freshly, 
locking  themselves  into  routine  behavior 
patterns.10 

Mental  models  serve  a  critical  function  in 
thinking  as  they  allow  individuals  to  give 
meaning  to,  and  thus  efficiently  process, 
what  otherwise  would  be  a  morass  of 
data.  But  they  can  cause  us  to  overlook, 
reject,  or  forget  important  incoming 
information  that  is  not  in  accord  with  our 
assumptions  and  expectations.  Mental 
models  strongly  influence  our  “search” 
processes  through  what  has  been 
called  “confirmation  bias,”  leading  us  to 
seek  information  that  supports  what  we 
believe  to  be  the  case  and  thus  having  a 
major  impact  on  research  that  underpins 
intelligence  analysis.* 11 


Mental  models  are  resistant  to  change, 
even  in  the  face  of  changing  external 
circumstances.  A  cognitive  psychologist 
observed  (at  one  of  the  project 
workshops)  that  in  analyzing  complex 
problems,  individuals  rely  on  “what 
has  worked  before”  and  rarely  update 
frameworks  (mental  models)  even 
when  they  can  no  longer  explain  new 
data.  This  is  because  humans  cannot 
change  their  way  of  thinking  very  quickly 
without  experiencing  a  disquieting 
sense  of  losing  control.  Experts  in  their 
fields,  whose  highly  developed  models 
allow  them  to  make  quick  and  accurate 
judgments  most  of  the  time,  are  often 
the  most  likely  to  cling  to  longstanding 
interpretations  in  the  face  of  anomalous 
information. 


This  is  the  “paradox  of  expertise”  that 
has  led  to  astonishing  misjudgments 
and  misguided  forecasts  by  bona 
fide  experts  in  science,  technology, 
and  business,  among  other  fields  of 
endeavor.  One  celebrated  example  is 
the  head  of  the  US  patent  office  at  the 
end  of  the  19th  century  who  declared 
confidently  that  virtually  everything  that 
could  be  invented  had  already  been 
invented.12 

Mental  models  are  not  the  only  factors 
that  can  inappropriately  influence 
thinking.  Others  include  “judgmental 
biases,”  such  as  a  common  tendency 
to  judge  the  probability  of  an  event 
by  the  “availability”  of  examples  of 
similar  types  of  event  rather  than  by 
its  mathematical  probability;  cultural 
biases,  such  as  a  belief  that  individuals 
in  other  cultures  will  act  or  react  similarly 
to  the  way  we  do;  and  motivational 
influences  on  thinking,  such  as  a  desire 
to  avoid  addressing  unpleasant  issues.13 
Because  such  deviations  from  purely 
rational  thinking  are  commonplace, 
remedial  steps  such  as  the  use  of 
alternative  analysis  techniques  are 
essential  to  avoid  misjudgments.14 

Obstacles  to  objective  thinking  affect 
not  only  the  individual  analyst  but 
larger  groups  as  well.  Students  of 
decisionmaking  have  documented  the 
phenomenon  of  “groupthink”  in  both 
public  and  private  organizations,  in 
which  the  desire  for  cohesion  within 
small  task  groups  whose  members 


10  Fred  I.  Greenstein  and  Michael  Lerner,  eds.,  A  Source  Book  for  the  Study  of  Personality  and  Politics  (Chicago: 
Markham,  1971). 

1 1  J.  Edward  Russo  and  Paul  J.H.  Schoemaker,  Winning  Decisions:  Getting  It  Right  the  First  Time,  (New  York: 
Doubleday,  2002),  pp.  84-85. 

12 

Found  on  Media  Futures  Archive ,  http://www.hfac.uh.edu/MediaFutures/home.html.  (last  visited 
23  April  2004. 

1j  An  exploration  of  these  factors  and  their  implications  for  intelligence  lies  beyond  the  scope  of  this  paper, 
although  we  will  address  a  few  later  on  in  the  context  of  discussing  transnational  alternative  analysis 

14  For  a  general  discussion  of  cognitive  biases  see  Ziva  Kunda,  Social  Cognition:  Making  Sense  of  People, 
(Cambridge:  MIT  Press,  2001 ),  pp.  53-1 1 0. 
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work  closely  together  over  time  imposes 
conformity  in  thinking.15  Even  more 
insidious  is  what  a  leading  thinker  on 
organizations  has  called  “organizational 
lock-in.”  A  mode  of  thinking  can  become 
so  dominant  in  an  organization’s  culture 
that  “confirmation  bias"  becomes 
embedded  in  its  decisionmaking 
system.  In  this  case,  the  organization 
as  a  whole,  not  just  individual  analysts 
or  decisionmakers,  primarily  searches 
for,  and  inevitably  finds,  information 
consonant  with  prevailing  ideas.16 

To  what  extent  in  practice  have  mental 
models  or  other  types  of  bias  negatively 
affected  intelligence  analysis?  In  fact, 
for  just  about  any  failure  in  warning, 
whether  American  or  foreign,  the 
contribution  of  cognitive  factors  is 
apparent. 

•  Prior  to  Pearl  Harbor,  for  instance, 
US  policymakers  held  assumptions 
and  expectations — that  it  would  be 
extremely  difficult,  if  not  impossible, 
for  Japan  to  attack  a  well  defended 
and  distant  naval  base — that 
contributed  to  the  lack  of  warning 
and  preparedness. 

•  The  Jeremiah  Commission 
blamed  the  alleged  failure  of  US 
intelligence  to  forecast  the  1997 
Indian  nuclear  test  on  cultural 
“mirror  imaging,"  which  led  analysts 
erroneously  to  believe  that  Indian 
leaders,  like  many  of  their  Western 


counterparts,  would  demur  on 
following  through  with  politically 
risky  campaign  pledges.17 

•  In  the  1973  Yom  Kippur  War,  to  take 
a  non-US  example,  Israeli  officials 
held  tenaciously  to  their  mental 
model  that  Egypt  and  Syria  would 
not  attack  until  they  had  regained 
the  potential  to  defeat  Israel, 
blinding  them  to  the  danger  of  an 
attack  with  more  limited  political 
objectives.18 

•  In  the  private  sector,  one  of 
the  most  dramatic  cases  was 
IBM’s  failure  in  the  mid-1980s 
to  appreciate  the  implications  of 
then  emerging  personal  computer 
technology  (a  field  which  it  could 
have  easily  dominated  given  its 
tremendous  financial  and  scientific 
resources),  reflecting  assumptions 
about  the  long-term  dominance  of 
mainframe  technology.19 

A  distant  but  still  instructive  warning 
failure  discussed  at  length  at  one 
workshop  was  that  of  French 
intelligence  in  1940,  which  was  unable 
to  comprehend  the  unorthodox  German 
tactic  of  invading  through  the  Ardennes 
Forest,  leading  to  the  defeat  of  the 
French  Army.20  French  intelligence, 
probably  the  best  in  the  world  at  that 
time,  undertook  sophisticated  analyses 
of  German  blitzkrieg  tactics  in  early 
1 940,  all  of  which  supported  the  belief 


1 5 

The  classic  work  is  Irving  L.  Janis,  Victims  of  Groupthink:  Psychological  Study  of  Foreign-Policy  Decisions 
and  Fiascoes  ( Boston:  Houghton  Mifflin,  1972). 

16  Kees  van  der  Heijden,  The  Sixth  Sense:  Accelerating  Organizational  Learning  with  Scenarios,  (New  York, 
John  Wiley,  2002),  pp.  50-51. 

The  post  mortem  was  chaired  by  former  vice  chairman  of  the  Joint  Chiefs  of  Staff,  Adm.  David  Jeremiah. 
The  report  was  never  made  public,  but  for  reportage  on  it,  see  the  Washington  Post,  June  3,  1998,  p.  A18  and 
the  New  York  Times,  same  date.  For  the  transcript  of  Jeremiah’s  briefing  on  the  report,  see  http://www.fas.ora/ 
irp/cia/product/jeremiah.html  (last  visited  16  December  2003). 

8  On  the  1 973  war,  see,  for  instance,  William  B.  Quandt,  Peace  Process:  American  Diplomacy  and  the  Arab- 
Israeli  Conflict  Since  1967,  (Washington,  Brookings  Institution  Press,  1990). 

19  Peter  Schwartz,  The  Art  of  the  Long  View,  (New  York,  Doubleday,  1991),  p.  220. 

20 

Ernest  R.  May,  Strange  Victory:  Hitler’s  Conquest  of  France,  (New  York:  Farrar,  Straus  and  Giroux,  2000). 
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that  the  brunt  of  an  attack  would  come 
through  the  plains  of  Belgium.  This 
increasingly  entrenched  view  actually 
helped  to  blind  the  French  general  staff 
to  mounting  evidence  of  an  Ardennes 
offensive,  such  as  aerial  photography 
of  German  pontoon  bridge  construction 
in  the  area  and  the  pattern  of  German 
reconnaissance  flights,  which  closely 
mirrored  the  later  invasion  route.  This 
example  demonstrates  that  even  “hard 
information" — often  touted  as  the  key  to 
improved  intelligence — may  be  of  little 
use  unless  it  is  received  by  “open  minds.” 


The  Nature  and  Role  of  Alternative 
Analysis 

“Alternative  analysis”  encompasses 
several  techniques  geared  toward 
broadening  an  analyst’s  thinking.  At  its 
most  basic,  alternative  analysis  can  take 
the  form  of  a  simple  “key  assumptions 
check”  by  which  an  analyst  explicitly 
states  the  underlying  beliefs  that  have 
been  guiding  analysis  of  an  issue  and 
then  seeks  to  evaluate  their  continued 
relevance.  More  sophisticated  forms 
include  “contrarian”  techniques  in 


Selected  Alternative  Analysis  Techniques 

“Contrarian” 

•  Devil’s  Advocacy  analysis  in  which  an  analyst  is  assigned  to  use  available  evidence 
to  develop  an  argument  contrary  to  the  prevailing  analytic  line. 

•  A  Team/B  Team  analysis  in  which  naturally  arising  differences  among  analysts 
are  brought  to  the  forefront  (rather  than  suppressed  as  in  “groupthink”)  by  dividing 
analysts  into  teams  tasked  with  producing  conflicting  interpretations  of  evidence. 

•  Red  Team  analysis  in  which  analysts  try  to  think  or  act  like  an  adversary  in  order  to 
test  prevailing  assumptions  about  “Red’s”  intentions  or  behavior. 

“Contingent” 

•  What  If?  analysis  in  which  an  analyst  varies  a  key  assumption  and  speculates  how 
this  might  lead  to  different  behavior  by  an  actor.  In  a  variant,  Low-Probability/High- 
Impact  analysis,  the  same  operation  is  done  but  with  the  aim  of  testing  an  unlikely 
but  plausible  outcome  with  dramatic  consequences  (what  if  poor  but  aggressive 
Ruritania  gets  hold  of  “the  bomb”). 

•  Alternative  scenarios  (called  scenario  planning  in  the  business  community)  involves 
a  more  complex  effort  to  conceive  two  or  more  plausible  alternative  outcomes  by 
identifying  and  then  exploring  the  interaction  of  underlying  drivers  of  behavior. 

The  outcomes  are  transformed  into  stories-hence  the  term  “scenarios-to  allow 
policymakers  to  more  deeply  understand  the  nature  and  implications  of  the 
alternative  outcomes. 
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which  prevailing  analytic  lines  and  their 
underlying  assumptions  undergo  direct 
challenge,  and  “contingent”  techniques 
that  broaden  the  range  of  outcomes 
considered  by  varying  underlying 
assumptions  (see  text  box).21  These 
techniques  may  be  carried  out  by 
individual  intelligence  analysts,  groups 
of  analysts,  or  outside  thinkers — alone 
or  in  combination  with  intelligence 
analysts — who  may  be  better  equipped 
than  insiders  to  bring  contrary  or 
divergent  perspectives  into  the  analytic 
process.22 

As  currently  applied  by  the  Intelligence 
Community,  alternative  analysis 
techniques  address  discrete  issues  and 
yield  “finished  intelligence.”  Issues  are 
specifically  selected  because  they  are 
subject  to  doubt  or  because  they  are 
deemed  too  important  to  “get  it  wrong.” 
A  fairly  intensive,  though  time  limited, 
effort  is  made  to  challenge  assumptions 
or  to  identify  alternative  outcomes, 
depending  on  the  technique  employed, 
culminating  in  a  written  product 
delivered  to  relevant  policymakers. 

This  approach  is,  on  the  one  hand,  a 
strength  of  alternative  analysis  because 
nothing  can  be  as  persuasive  as  a  well 
researched,  well-argued  written  product 
or  briefing  delivered  at  the  right  time 
to  a  policymaker.  But  for  amorphous, 
continuous  threats,  for  which  there 


may  be  no  “right  time,”  the  specificity 
and  product-driven  focus  of  alternative 
analysis  can  be  more  problematic. 

The  degree  to  which  alternative  analysis 
has  been  effective  in  addressing  the 
challenges  of  uncertainty  and  mental 
biases  in  the  intelligence  sector  is 
unclear,  particularly  given  the  spotty 
application  of  these  techniques  to 
date.  In  the  business  sector,  where  the 
technique  of  scenario  planning  has 
been  used  more  systematically  over  a 
sustained  period,  there  appear  to  have 
been  some  successes.  Royal  Dutch 
Shell,  for  example,  has  used  scenario 
planning  for  some  30  years  to  anticipate 
and  react  to  major  discontinuities 
in  energy  markets  (such  as  the  oil 
embargoes  of  the  1970s),  transforming 
itself  in  the  process  into  one  of  the 
larger  of  the  major  global  energy 
companies.23 
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Shell  is  far  and  away  the  most  often 
mentioned  success  story,  but  scenarios 
are  used  at  least  periodically  by  an 
estimated  one  third  of  US  companies — 
presumptive  evidence  that  bottom-line 
focused  private-sector  executives  find 
them  helpful.24 


The  potential  gains  from  employing 
alternative  analysis  come  at  some  cost, 
as  raised  in  workshop  discussions. 


We  should  note  that  in  addition  to  the  above,  what  are  called  “serious-play”  techniques — military  and  political 
gaming,  computer  simulation — are  sometimes  included  within  the  alternative  analysis  rubric.  These  techniques 
test  assumptions  interactively  by  having  humans  or  intelligent  agents  play  out  situations  implied  by  those 
assumptions.  However,  “serious  play”  stands  apart  from  the  alternative  analysis  mainstream  in  that  it  involves 
a  much  higher  level  of  methodological  sophistication  and  resource  commitment  than  is  usually  the  case  with 
contrarian  or  contingent  techniques.  Although  a  few  interesting  ideas  emerged  related  to  gaming  and  simulation 
during  the  course  of  workshop  discussions  (and  will  be  reported  later  in  this  paper),  we  did  not  explore  this  very 
specialized  area  at  any  length. 

For  an  overview  of  alternative  analysis  techniques,  see  Roger  George,  “Fixing  the  Problem  of  Analytical 
Mind-Sets:  Alternative  Analysis,”  International  Journal  of  Intelligence  and  Counter-Intelligence,  Vol.  1 7,  No.  3, 

Fall  2004,  pp.  385-405. 

23 

For  a  brief  description  of  Shell’s  scenario  process,  see  P.  Schoemaker  and  K.  van  der  Heijden,  “Integrating 
Scenarios  into  Planning  at  Royal  Dutch/Shell,”  Planning  Review,  20,  3  (1992). 

24 

“  Kris  Frieswick,  “The  Difficulties  of  Thinking  Ahead,”  in  CFO.com,  February  1 , 2002,  http://www.cfo.com.  last 
viewed  4  May  2004. 
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•  Alternative  analysis  efforts 
sometimes  take  analysts  off-line, 
risking  slip-ups  in  current  coverage. 

•  Contrarian  techniques  can 
undermine  group  cohesion  and, 
on  controversial  issues,  where 
alternative  views  are  associated 
with  policymaker  positions,  can 
present  the  appearance  that 
analysis  has  been  “politicized” — 
shaped  to  produce  a  particular 
answer. 

•  Alternative  analysis  directs 
attention  to  outcomes  that,  while 
possible,  are  almost  by  definition 
improbable,  potentially  diverting 
policy  attention  and  resources  away 
from  more  likely  threats.  Harried 
policymakers  looking  for  “answers” 
may  not  understand  or  appreciate 
the  subtle  arguments  that  emerge 
from  the  alternative  analysis 
process,  and  may  be  tempted  to 
consider  it  a  form  of  analytic  self- 
defense. 

These  downsides,  while  important 
to  keep  in  mind,  do  not  outweigh  the 
gains  possible  from  utilizing  these 
techniques,  however,  particularly  given 
the  potential  catastrophic  consequences 
of  intelligence  misjudgments. 

Looking  at  Transnational  Issues: 
How  Are  They  Different? 

A  key  issue  examined  as  part  of 
the  project  was  the  extent  to  which 
transnational  issues  differ,  as  an 
intelligence  challenge,  from  more 
traditional  state-to-state  issues. 

The  prevailing  view  among  project 
participants — who  included  analysts 
with  broad  experience  dealing  with  both 
types  of  issues — was  that  significant 


differences  do  exist.25 This  judgment, 
it  must  be  noted,  appeared  to  be 
predicated  on  an  implicit  comparison  of 
the  dominant  threats  of  recent  years — 
the  Soviet  Union  and  Al-Qa‘ida — rather 
than  a  more  systematic  consideration 
of  the  full  range  of  transnational  and 
traditional  issues.  Moreover,  some 
participants  argued  that  there  was 
considerable  overlap  between  state- 
to-state  and  transnational  issues, 
suggesting  the  need  to  consider  a 
crosscutting  taxonomy  of  intelligence 
issues  (as  will  be  explored  later  on  in 
this  section).  That  said,  we  present 
below  participants’  impressions  of 
the  key  differences  because  they 
offer  insights  into  the  types  of  special 
challenges  faced  in  important  segments 
of  the  transnational  realm. 

In  comparison  to  their  state-to-state 
counterparts,  transnational  intelligence 
issues  were  said  to: 

•  Be  less  “bounded.” 

•  Exhibit  a  closer  action-reaction 
relationship — or  “observer-actor” 
linkage — between  key  players. 

•  Unfold  more  quickly  and 
suddenly. 

•  Present  a  broader,  but  lower- 
quality  information  base. 

Resulting  in: 

•  Requiring  more  collaborative  and 
multidisciplinary  analysis. 

•  Having  to  support  a  broader  range 
of  consumers. 

Less  “bounded”:  States  are,  for  the 
most  part,  clearly  “delineated,”  having 
known  borders  and  capitals,  and 


25 

The  project  focused  on  three  transnational  issues — terrorism,  weapons  proliferation,  and  organized  crime — 
with  experts  in  these  fields  offering  opening  presentations  at  the  first  workshop.  In  practice,  terrorism  tended  to 
drive  workshop  discussions,  being  uppermost  in  the  minds  of  most  participants. 
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doing  much  that  is  openly  observable. 
Moreover,  states  act  within  the  context 
of  formal  and  customary  rules,  such 
as  established  military  doctrines, 
international  law,  and  historical 
precedents,  giving  predictability  to  many 
of  their  actions.  Transnational  actors 
are  amorphous,  fluid,  and  hidden, 
presenting  intelligence  analysts  with 
major  challenges  simply  in  describing 
their  structures  and  boundaries.  And 
because  such  actors  are  also  far  less 
constrained  by  formal  rules  than  their 
state  counterparts,  they  can  engage 
in  a  wider  variety  of  tactics  on  a 
regular  basis,  adding  immensely  to  the 
challenge  of  forecasting  their  behavior. 

Closer  observer-actor  linkage:  The 

former  Soviet  Union  was  a  relatively 
independent  actor,  with  much  of  its 
behavior  driven  by  internal  factors 
rather  than  by  the  international  system. 
(Recall  the  famous  line  attributed  to  a 
US  defense  secretary  about  the  nuclear 
relationship  between  the  United  States 
and  the  Soviet  Union:  when  we  build 
up,  they  build  up;  when  we  slow  down, 
they  build  up.)  By  contrast,  transnational 
actors  have  a  more  intense  relationship 
with  the  dominant  actor  in  the 
international  system,  the  United  States. 
Their  tactics  are  often  predicated  upon 
our  policies  and  defensive  measures, 
making  their  behavior  less  determinate 
and  predictable.  Our  understanding  of 
transnational  actors’ intentions  will  lead 
us  to  take  actions  that  will — to  a  greater 
extent  than  would  be  the  case  with 
more  structured,  internally  driven  state 
actors — prompt  adaptive  behavior  on 
their  part.  This  will  turn  predictions  into 
“self-negating  prophecies.” 


Events  unfold  more  quickly  and 
suddenly:  Intelligence  analysts  had 
a  decade  to  explore  the  impact  of 
Gorbachev’s  accession  on  the  Soviet 
system,  according  to  workshop 
participants,  but  in  the  case  of  Al  Qa'ida, 
for  example,  events  have  unfolded  at  a 
stunning  pace  since  the  August  1998 
attacks  on  US  Embassies  in  Kenya 
and  Tanzania.  Generally  speaking, 
as  flexible  and  adaptive  entities, 
transnational  actors  are  more  capable 
than  most  governmental  actors  of 
engaging  in  sudden  shifts  of  behavior. 
Moreover,  tightly  linked  transnational 
networks  are  subject  to  what  network 
scientists  refer  to  as  “cascades,"  which 
are  fast,  domino-like  sequences  of 
events  that  can  result  from  small 
perturbations  (as  in  a  cascading  electric 
power  grid  failure).  To  take  a  vivid 
example  from  the  economic  domain,  a 
relatively  small  crisis  involving  the  Thai 
currency  quickly  enveloped  much  of 
Asia  in  1997,  as  transnationally-linked 
financial  markets  suffered  cascading 
withdrawals  of  funds  and  confidence.26 


. . .  transnational 
actors  are  more 
capable . . . 
of  engaging  in 
sudden  shifts  of 
behavior. 


Broader  and  lower-quality 
information  base:  Because  of  the 
unbounded  and  high-profile  nature  of 
transnational  threats,  analysts  must 
wade  through  a  sea  of  information  that 
contrasts  sharply  with  the  much  more 
limited  information  available  on  closed 
societies  such  as  the  Soviet  Union. 

And  much  of  the  information  is,  at 
best,  of  uncertain  reliability.  Moreover, 
as  compared  with  a  state  with  a  long 
history,  much  less  contextual  information 
is  available  that  can  be  used  to  evaluate 
the  reliability  of  new  information. 

For  these  reasons,  the  problem  of 
separating  “signals”  from  “noise”  is  more 
acute  in  the  transnational  domain,  as 
is,  arguably,  the  problem  of  detecting 
deception. 


For  a  readable  discussion  of  networked  phenomena  see  Albert-Laszlo  Barabasi,  Linked:  How  Everything  is 
Connected  to  Everything  Else  and  What  It  Means,  (New  York:  Plume,  2003). 
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..  .the  challenge 
of  ‘connecting’ 
substantively— 
conceptually  and 
contextually — 
with  the 
consumer  is 
greater  in  the 
transnational 
arena. 


More  collaborative  and 
interdisciplinary:  While  a  country 
political  or  economic  analyst  can  often 
work  in  relative  isolation  from  analysts 
with  other  specializations,  transnational 
issues  require  combinations  of  regional 
and  functional  expertise.  Weapons 
proliferation  analysis,  for  instance, 
draws  upon  specialists  in  science  and 
technology,  illicit  transfers,  money 
laundering,  politics,  and  network 
behavior,  to  name  but  a  few,  to  track  and 
comprehend  the  activities  of  weapons 
networks.  To  a  much  greater  extent 
than  in  traditional  areas,  transnational 
analysis  is  a  team  or  even  networked 
activity  (as  specialists  will  be  located  in 
many  agencies).  This  has  both  potential 
benefits  in  terms  of  avoiding  mental 
biases  (mixing  different  perspectives) 
but  also  potential  risks  in  the  form  of 
groupthink  and  “lock-in,”  to  say  nothing 
of  the  inevitable  practical  difficulties 
of  ensuring  robust  communication 
and  information-sharing  among 
diverse  parties  across  organizational 
boundaries. 


Consumer  base  is  different:  For 

state-to-state  issues,  key  consumers 
are  relatively  few,  most  of  them  located 
at  the  apex  of  the  national  security 
decisionmaking  establishments. 
Intelligence  and  its  consumers  also 
share  a  common  “story”  in  dealing  with 
states.  Both  know  that  states,  even 
very  different  states,  are  territorial  and 
usually  hierarchical,  with  histories, 
traditions  and  standard  operating 
procedures.  All  that  is  much  less  true 
for  transnational  issues  and  actors, 
in  part  because  they  are  relatively 
new  phenomena.  We  have,  at  best,  a 
provisional  story  about  them,  or,  more 
likely,  tentative,  competing  stories  about 
them  based  on  often  short  histories. 

For  this  reason,  too,  the  challenge 
of  “connecting”  substantively — 
conceptually  and  contextually — with  the 


consumer  is  greater  in  the  transnational 
than  in  the  state-to-state  arena.  For 
transnational  issues,  the  number  and 
variety  of  consumers  is  also  much 
greater,  extending  to  a  variety  of 
agencies  outside  the  traditional  national 
security  establishment.  Moreover,  it  is  far 
less  clear  who  is  the  “key”  consumer.  An 
airport  security  officer  or  a  public  health 
doctor  may  have  a  more  urgent  “need  to 
know”  about  a  threat  than  the  President 
because  he  or  she  may  be  in  a  more 
immediate  position  to  thwart  it.  At  the 
same  time,  most  new  transnational 
consumers  have  little  familiarity  with 
intelligence  and  how  to  interpret  it. 

Beyond  State  and  Transnational: 
Seeking  a  Better  Taxonomy 

Some  workshop  participants,  as  noted 
before,  argued  that  the  domain  of 
transnational  issues  could  not  easily 
be  separated  from  state-to-state 
issues.  Some  transnational  issues, 
such  as  weapons  of  mass  destruction 
(WMD)  proliferation,  feature  significant 
involvement  of  state  as  well  as 
transnational  actors.  Moreover,  the 
transnational  arena  contains  entities 
that  are  as  bounded  and  stable  as 
nation-states,  such  as  organized  crime 
syndicates.  At  the  same  time,  within 
the  state-to-state  arena,  there  are 
key  intelligence  issues  that,  like  many 
transnational  issues,  are  unbounded, 
fast  moving,  characterized  by  adaptive 
behavior,  and  obscured  by  voluminous 
information,  such  as  crisis  diplomacy, 
failed  states,  insurgencies  or  the 
battlefield  environment.  The  challenges 
facing  analysts  in  comprehending  al 
Qa'ida  may  not,  in  fact,  be  that  much 
different  from  those  confronting  analysts 
in  the  runup  to  the  Battle  of  France  or 
Pearl  Harbor — even  as  they  probably 
are  different  from  those  associated  with 
tracking  fairly  consistent  Soviet  behavior 
during  the  latter  stages  of  the  Cold  War. 
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Table  1 :  Traditional  vs.  Transnational  Targets 


Traditional  Targets 

Transnational  Targets 

Focus:  states,  nonstates  secondary 

Nonstates;  states  as  facilitators,  willingly  or 
not 

Nature  of  targets:  hierarchical 

Networked 

Context:  intelligence  and  policy  share 
basic  “story”  about  states 

Much  less  shared  story  about  nonstates, 
less  “bounded,”  more  outcomes  possible 

Information:  too  little  information,  pride 
of  place  to  secrets,  secrets  regarded  as 
reliable 

Secrets  matter,  but  torrents  of  information, 
fragmented,  unreliable 

Pace  and  trajectory  of  events:  primary 
target  slow  moving,  discontinuities  rare 

Targets  may  move  quickly,  discontinuities  all 
too  possible 

Interaction  effects:  limited 

“Your”  actions  and  observations  have  more 
effect  on  target’s  behavior 

Need  for  collaboration:  limited,  analysis  in 
“stovepipes” 

Greater  with  both  regional  and  functional 
intelligence  specialists,  plus  different  levels 
of  government 

Policy  support:  consumers  mostly  politico- 
military  officials  of  federal  government 

Wider  range  of  consumers,  intelligence 
often  linked  to  action  on  a  continuing  basis 

. . .  discussions 
focused  on 
identifying  an 
underlying 
category . . . 

which  can 
include  both 
transnational  and 
state  issues. 


Given  overlap  between  transnational 
and  traditional  (or  state)  intelligence 
issues,  workshop  discussions 
focused  on  identifying  an  underlying 
category  displaying  the  problematic 
characteristics  cited  above — unbounded, 
fast  moving,  and  the  like — but  which 
can  include  both  transnational  and 
state  issues.  One  promising,  cross¬ 
cutting  framework  was  the  division  of 
problems  into  “ordered”  and  “disordered” 
categories,  as  developed  to  address 
business  decisionmaking  issues  by  the 
management  scientist  David  Snowden.27 
Ordered  problems  follow  established 
patterns  and  thus  can  be  understood 
through  rational  means  of  inquiry; 
disordered  problems  do  not  follow  such 
patterns  and  thus  are  not  amenable  to 
reductionist  techniques.  Snowden  further 


divides  ordered  space  into  “known”  and 
“knowable”  problems,  and  disordered 
space  into  “complex”  and  “chaotic” 
problems. 

•  Known  problems  are  those 
for  which  there  is  a  unique 
deterministic  relationship  between 
causes  and  effects;  the  challenge  is 
to  correctly  categorize  the  problem, 
obtain  the  necessary  data  to  solve 
it,  and  apply  accepted  formulas. 

An  example  is  a  process¬ 
engineering  issue  for  which  there 
are  “best-practice”  solutions. 

•  Knowable  problems  involve 
contingent  relationships  between 
a  limited  set  of  causes  and  effects. 
In  this  realm  it  is  possible  to  use 


27 

David  Snowden,  “Complex  Acts  of  Knowing:  Paradox  and  Descriptive  Self-Awareness,”  Journal  of  Knowledge 
Management,  Special  Issue,  September  2002,  available  at  http :// www. kwork.org/Resou rces/snowden  .pdf. 

(last  visited  December  17,  2003).  Snowden’s  framework  derives  from  complexity  science,  which  examines 
inter-relationships  in  systems  whose  units  are  interconnected  and  where  the  entire  system  exhibits  properties 
different  from  the  sum  of  its  parts.  For  a  general  discussion  of  the  implications  of  complexity  for  international 
systems,  see  Robert  Jervis,  System  Effects'.  Complexity  in  Political  and  Social  Life,  (Princeton:  Princeton 
University  Press,  1997). 
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analytic  reductionist  techniques 
to  predict  outcomes,  at  least 
probabilistically.  For  example,  an 
investment  decision  in  an  existing 
business  area  lies  within  this  realm 
as  one  can  usually  identify  the  limited 
set  of  factors  (market  demand, 
strategies  of  the  competition,  and 
so  on)  that  will  influence  the  set  of 
expected  outcomes. 

•  Complex  problems  involve  a  wide 
array  of  causes  and  effects  that  can 
interact  in  a  variety  of  contingent 
ways.  This  is  the  realm  of  interactive 
systems  such  as  battlefields, 
ecosystems,  and  competitive 
markets  in  which  large  numbers  of 
relatively  small  actors  respond  to 
a  shifting  set  of  situational  factors. 


Because  interactions  reflect 
unique  circumstances,  they  do 
not  necessarily  repeat  in  any 
established  pattern  and  are  not 
amenable  to  predictive  analysis. 
The  best  that  can  be  done  in  the 
case  of  such  problems  is  to  “sense” 
emerging  patterns  with  an  eye  to 
reinforcing,  disrupting,  or  taking 
advantage  of  them. 

•  Finally,  there  are  chaotic  problems 
where  there  is  no  discernible 
relationship  between  causes  and 
effects.  Such  problems  (e.g.  the 
aftermath  of  a  natural  disaster) 
require  immediate  action  to  bring 
them  to  a  more  ordered  state 
where  reflective  thought  processes, 
including  those  used  in  intelligence, 
can  be  effective. 


Puzzles  and  Mysteries 

A  puzzle  is  a  problem  for  which  there  is  a  solution  in  principle,  if  only  the  right  information 
could  be  found  (thus  comparable  to  Snowden’s  “known’  problems).  A  puzzle  in 
intelligence  terms  is  primarily  a  challenge  to  collection;  the  “counting”  issues  of  the  Cold 
War — how  many  missiles  did  the  Soviets  have,  and  how  accurate  were  they? — are  classic 
examples  of  this  genre.  So  was  whether  Iraq  had  weapons  of  mass  destruction. 

A  mystery,  by  contrast,  is  a  problem  for  which  there  can  be  several  outcomes  depending 
upon  how  underlying  driving  forces  combine  (thus  comparable  to  Snowden’s  “’’knowable” 
problems).  Mysteries  are  questions  without  a  certain  answer,  even  in  principle,  because 
they  are  future  and  contingent.  Collection  is  less  crucial  in  this  realm  because  information 
can  only  provide  clues  as  to  the  likelihood  of  outcomes,  not  a  definitive  answer.  The  future 
of  Chinese  governance  is  an  example  of  a  mystery:  it  may  be  democratic,  authoritarian, 
or  something  in  between  and  no  amount  of  information  will  prove  diagnostic.  Will  North 
Korea  dismantle  its  nuclear  weapons?  What  will  Russia’s  inflation  rate  be  next  year? 
These,  too,  are  mysteries,  susceptible  at  best  to  probabilistic  prediction. 

The  relevance  of  the  puzzle  and  mystery  metaphors  to  transnational  issues  came  up 
repeatedly  in  workshop  discussions.  Some  argued  that  these  issues  were  puzzles  but 
with  many  of  the  key  pieces  missing.  Yet  it  is  unclear  whether  a  puzzle  that  is  essentially 
insoluble  can  usefully  be  thought  of  in  these  terms.  The  mystery  metaphor  also  is 
problematic  as  it  implies  that  one  can  at  least  assess  likeliest  outcomes  well  in  advance 
by  carefully  evaluating  available  evidence.  This  may  work  for  broad  strategic  trends  of 
known  transnational  actors — future  directions  of  Al-Qa'ida  or  a  criminal  syndicate — just 
as  it  does  for  a  country,  but  it  does  not  seem  applicable  to  tactical-level  activities  of 
such  actors,  which  include  a  very  wide  range  of  possibilities.  And  it  also  does  not  seem 
applicable  to  understanding  developments  in  the  more  inchoate,  less  visible  segments  of 
the  transnational  realm  such  as  those  involving  emerging  or  loosely  structured  networks 
(e.g.  jihadist  groups).  It  is  arguably  at  these  levels  where  the  most  significant  warning 
challenges  reside. 
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The  Snowden  framework  suggests  that 
there  is  a  class  of  problems  beyond  the 
well-established  division  of  intelligence 
issues  into  “puzzles”  and  “mysteries” 
(see  text  box)  that,  for  want  of  a  better 
term,  can  be  called  “complexities.”28 
These  problems  involve  some 
combination  of  the  following  factors: 
large  numbers  of  small  sized  actors, 
fluidity  of  rules  governing  behavior,  and 
the  large  influence  of  situational  as 
opposed  to  internal  factors  in  shaping 
behavior.  Due  to  these  characteristics, 
these  problems  can  yield  a  wide  range 
of  sui  generis  outcomes  that  defy 
probabilistic  prediction. 

Complex  problems  are  prevalent  in  the 
transnational  realm  because  actors 
are  small,  numerous,  and  relatively 
unbounded  by  rules,  and  processes  are 
highly  interactive.  From  the  perspective 
of  the  intelligence  analyst,  the  presence 
of  overwhelming  amounts  of  information 
of  questionable  reliability  adds  to  this 
complexity  because  it  is  difficult  even  to 
ascertain  current  realities. 

To  be  sure,  complex  problems  also 
describe  some  state-to-state  intelligence 
issues,  such  as  crisis  diplomacy  and 
battlefield  intelligence  where  high 
interactivity  and  information  overload 
(e.g.  the  “fog  of  war”)  are  present.  One 
major  difference,  however,  is  that  crisis 
or  battlefield  conditions  are  usually 
time-limited  or  exceptional  in  the  state- 
to-state  realm,  whereas  equivalent 
conditions  are  an  ongoing  fact  of  life  in 
the  transnational  realm. 


Understanding  Complex  Issues: 
Intuitive  Thinking  and  the  Concept 
of  “Sense-Making” 

If  “complexities”  are  resistant  to 
reductionism  and  prediction,  then 
it  is  important  to  explore  how  such 
problems  are  or  can  be  addressed 
by  intelligence  in  order  to  identify  an 
appropriate  “alternative”  process  for 
tackling  mental  biases.  To  do  so,  it  is 
useful  to  begin  by  reconsidering  the 
concept  of  “intelligence  analysis.”29 
Upon  close  inspection,  this  concept 
actually  encompasses  not  only 
“analysis,”  strictly  defined,  but  also  other 
mental  processes  that  are  actually 
more  relevant  to  the  comprehension 
of  complex  problems  encountered  in 
intelligence  issues. 

Analysis,  according  to  one  reasonable 
definition,  involves  breaking  down  a 
problem  into  its  constituent  parts  and 
then  using  logical  processes  to  derive 
useful  conclusions,  whether  this  is 
for  the  purposes  of  explanation  or 
prediction. Those  logical  processes 
include  generating  a  range  of  plausible 
hypotheses  and  then  rigorously 
evaluating  the  hypotheses  in  light 
of  available  evidence  according  to 
accepted  criteria.30  According  to  Heuer, 
such  criteria  may  be: 

•  Situational  (sui  generis  factors 
intrinsic  to  the  situation  under  study, 
as  is  used  in  narrative  history); 


On  the  distinction  between  puzzles  and  mysteries,  see  Gregory  F.  Treverton,  “Estimating  Beyond  the  Cold 
War,”  Defense  Intelligence  Journal ,  3,  2  (Fall  1994);  and  Joseph  S.  Nye,  Jr. ,  “Peering  into  the  Future,”  Foreign 
Affairs,  77,  4  July/August  1994,  82-93. 

29 

The  modern  father  of  intelligence_analysis  is  Sherman  Kent.  See  his  Strategic  Intelligence  for  American 
World  Policy,  (Princeton,  NJ:  Princeton  University  Press,  1949).  For  a  wonderful  review  of  Kent’s  book,  one 
that  particularly  focused  on  two  shortcomings  in  traditional  foreign  intelligence  analysis,  see  Willmoore  Kendall, 
“The  Function  of  Intelligence,”  World  Politics,  1,  6  (July  1949),  542-52.  Given  the  bright  white  line  between 
intelligence  and  policy,  intelligence  usually  analyzed  foreign  government  in  isolation  (or  ignorance)  of  what  ours 
was  doing,  and,  related,  most  intelligence  was  limited  to  “foreign”  developments  and  so  found  it  difficult  (and 
perhaps  illegal)  to  assess  the  impact  of,  say,  American  banks  on  Asia’s  finances. 

30 

“  This  definition  is  a  composite  of  several  definitions  of  analysis  discussed  in  Rob  Johnston,  “Developing 
a  Taxonomy  of  Intelligence  Analysis  Variables,”  Studies  in  Intelligence,  Vol.  47,  No.  3,  2003  found  at  http:// 
www.cia.aov.  last  viewed  May  4,  2004. 


. .  there  is  a  class 
of  problems . . . 
that  can  be  called 
“complexities.” 
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. . .  good  analysis 
. . .  should 
focus  more  on 
disconfirming 
rather  than 
confirming 
evidence . . . 


Heuer  further  argues  that  good 
analysis  should  satisfy  an  even  more 
stringent  requirement.  As  in  scientific 
investigation,  it  should  focus  more  on 
disconfirming  rather  than  confirming 
evidence,  because  a  hypothesis  that  is 
supported  by  evidence  may,  in  fact,  be 
one  of  several  in  that  category,  in  which 
case  the  evidence  is  not  diagnostic, 
while  a  hypothesis  that  is  disconfirmed 
can  be  categorically  eliminated  from 
consideration. 

Heuer  maintains,  based  on  his  own 
experience  and  observations,  that 
intelligence  analysis  does  not,  in 
practice,  really  fit  any  stringent  criteria 
of  analysis.  It  tends  to  be  a  relatively 
informal  process  of  reaching  judgments 
based  on  confirmatory  evidence  in 
which  the  aim  is  to  “satisfice” — to 
present  the  minimally  best  case  that 
will  satisfy  consumers  or  superiors.  Rob 
Johnston,  in  a  very  recent  unclassified 
survey  of  analytic  practice  throughout 
the  Intelligence  Community,  came  to 
broadly  similar  conclusions.31  He  argues 
that  most  of  what  passes  for  intelligence 
analysis  is  reportorial — ’’CNN  plus 
secrets” — coupled  with  general  reflection 
and  discussion  that  uses  the  prevailing 


•  Deductive  (generalizations  about 
human  behavior  drawn  from  social 
scientific  investigation); 

•  Or  by  analogy  (comparing  the 
present  situation  to  historically 
similar  situations  with  an  eye  to 
comparing  the  validity  of  cause-effect 
relationships). 


line  as  a  starting  point.  Formal  analytic 
tools  (Johnston  counts  about  160,  most 
of  which,  however,  have  been  developed 
outside  the  intelligence  realm)  are 
used  sparingly  and  haphazardly  with 
no  common  doctrine  about  their  use 
within,  much  less  across,  intelligence 
agencies.32  Johnston  attributes  this 
state  of  affairs  to  the  unfamiliarity 
of  subject  matter  experts  with 
methodological  tools,  and  cites  the 
need  to  bring  analytic  methodologists 
into  the  intelligence  process  to  promote 
the  employment  of  formal  analytic 
techniques. 

Heuer’s  and  Johnston’s  observations 
suggest  that  intelligence  analysts 
engage  in  mental  processes  other  than 
analysis,  strictly  conceived.  To  define 
these  residual  processes,  it  is  useful  to 
draw  upon  several  constructs  developed 
by  management  scientists  to  categorize 
less  structured  forms  of  perception  and 
decisionmaking  that  take  place  within 
business  organizations. 

•  One  of  these  is  the  concept 
of  intuitive  decisionmaking, 
particularly  as  developed  by 
such  thinkers  as  management 
expert  Gary  Klein  and  cognitive 
psychologist  Guy  Claxton.33 

•  A  second  concept  is  that  of 
organizational  “sense-making,” — 
essentially  intuition  at  the 
organizational  level — as  developed 
by  the  noted  organization  theorist, 
Karl  Weick.34 


Johnston  is  an  anthropologist  by  training.  See  Rob  Johnston,  The  Culture  of  Analytic  Tradecraft  An 
Ethnography  of  the  Intelligence  Community  (Washington:  Center  for  the  Study  of  Intelligence,  Central 
Intelligence  Agency),  in  press. 

op 

Such  ideas  raised  heated  discussions  at  one  of  the  GFP/RAND  workshops.  Some  participants  argued 
that  analytic  offices  in  the  Intelligence  Community  were  employing  increasingly  sophisticated  methodologies 
to  address  complex  questions,  although  others  argued  that  this  is  not  the  same  thing  as  regularly  applying 
structured  techniques  to  day-to-day  issues. 

33 

Gary  Klein,  Intuition  at  Work:  Why  Developing  Your  Gut  Instinct  Will  Make  You  Better  at  What  You  Do,  (New 
York:  Doubleday,  2002). 

See  his  Sensemaking  in  Organizations,  (London:  Sage  Publications,  1995). 
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Klein  argues,  based  on  studies  of 
real  world  decisionmaking  among 
businessmen,  firefighters,  and  military 
officers,  that  most  people  base 
decisions  on  intuition  or  “gut  instinct” 
rather  than  upon  formal  analysis.  This 
process  (more  formally  called  response- 
primed  decisionmaking)  involves  the 
relatively  unstructured  application  of 
subject  expertise  and  overall  experience 
to  detect  patterns  of  opportunity  and 
threat,  which  in  turn  shape  action. 

Intuitive  decisionmaking  works  in 
areas  marked  by  well-defined  and 
repetitive  problems  (such  as  medical 
diagnosis)  as  well  as  in  more  uncertain 
areas  where  the  “gut”  provides  useful 
guidance.  As  one  popular  business 
writer  has  pointed  out,  commodity 
traders,  who  develop  well  honed  gut 
instincts  to  survive  in  a  fast  moving, 
highly  uncertain  market  environment, 
have  been  able  to  beat  more  analytically 
focused  professionals  such  as  US 
Marines  in  simulated  competitions, 
even  in  the  latter’s  own  game.35  Indeed, 
clever  defense  lawyers  seek  to  discredit 
ballistics  experts  by  requiring  them  to 
describe  the  analytic  steps  through 
which  they  reached  their  conclusions. 
They  do  this  knowing  that,  like  chess 
masters,  those  experts  reach  their 
conclusions  not  through  a  strictly 
analytic  process  but  rather  through  a 
difficult  to  articulate  process  of  patterns 
as  much  sensed  as  analyzed — the 
results  of  having  seen  thousands  of 
previous  cases. 

Traditional  analysis,  according  to 
Klein,  complements  intuition  where 
there  is  a  structured  problem  to  be 
addressed  and  time  for  serious  study. 
But  attempts  to  substitute  analysis  for 


intuition  in  situations  marked  by  high 
uncertainty  can  be  counterproductive, 
as  the  reductionism  needed  to  fit 
such  problems  into  models  or  highly 
structured  analytic  schemes  may 
produce  misleading  results. 

In  another  take  on  intuition,  psychologist 
Guy  Claxton  lays  out  two  processes 
of  thinking — a  “deliberative”  mode  that 
corresponds  to  conscious  analytic 
thinking  and  a  more  “contemplative” 
mode  that  corresponds  to  intuitive, 
subconscious  thinking.36 

•  Deliberative  thinking  is  focused 
on  answers,  solutions,  and  logical 
explanations;  is  purposeful,  effortful, 
and  fast;  is  reliant  upon  literal  and 
explicit  language;  and  works  well 
when  tackling  problems  that  can 

be  treated  as  an  “assemblage  of 
namable  parts.” 

•  Contemplative  thinking  is 
focused  more  on  questions  and 
observations,  is  playful,  metaphoric, 
imprecise,  and  slow  moving,  and 
works  best  on  problems  that  must 
be  grasped  holistically. 

Claxton’s  concept  of  intuitive  thinking 
differs  in  some  key  respects  from 
Klein’s — it  is  slow  instead  of  fast 
moving — but  his  conclusions  about 
its  effectiveness  in  addressing  issues 
marked  by  high  uncertainty  are  similar. 
He  writes  that  “recent  scientific  evidence 
shows  convincingly  that  the  more 
patient,  less  deliberative  modes  of  mind 
are  particularly  suited  to  making  sense 
of  situations  that  are  intricate,  shadowy 
or  ill  defined.”  Such  situations  would 
seem  to  correspond  to  the  “complexities” 
of  transnational  and  other  issues. 


. . .  intuition  or 
‘gut  instinct’ . . . 
involves  the 
relatively 
unstructured 
application  of 
subject  expertise 
and  overall 
experience  to 
detect  patterns  of 
opportunity  and 
threat . . . 

. . .  attempts  to 
substitute  analysis 
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. . .  ‘the  more 
patient,  less 
deliberative 
modes  of  mind 
are  particularly 
suited  to 
making  sense  of 
situations  that 
are  intricate, 
shadowy  or  ill 
defined.’ 


35 

Thomas  Stewart,  “How  To  Think  with  Your  Gut"  in  Business  2.0,  November  2002,  http://www.business2.0.com. 
According  to  Stewart,  the  Marines  moved  to  a  more  intuitive  approach  to  decisionmaking  training  partly  in 
response  to  this  experience. 

Guy  Claxton,  Hare  Brain,  Tortoise  Mind:  How  Intelligence  Increases  When  You  Think  Less,  (New  York:  Harper 
Collins,  2000),  pp.  3-11. 
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Intelligence 

sense-making 

offers  a  way  to 

comprehend 

‘complexities,’ 

whose 

combination 

of  speed, 

uncertainty,  and 

interactivity 

defy  traditional 

analytic 

approaches. 


Weick’s  concept  of  sense-making 
involves  the  process  through  which 
organizations — vice  individuals — 
comprehend  the  complex  environment 
with  which  they  must  contend.  It  is  a 
continuous,  iterative,  largely  informal 
effort  to  understand,  or  “make  sense” 
of  what  is  going  on  in  the  external 
environment  that  is  relevant  to  the 
organization’s  goals  and  needs.  In 
essence,  it  is  the  collective  intuition  of 
an  organization.  Through  conversations 
at  all  levels,  organizations  construct 
ongoing  interpretations  of  reality 
by  comparing  new  events  to  past 
patterns,  or  in  the  case  of  anomalies, 
by  developing  stories  to  account  for 
them.  Weick  argues  that  the  fluid  sense¬ 
making  process  has  clear  advantages 
as  a  framework  for  organizational  action 
over  “decisionmaking”  because  the 
latter  often  locks  the  organization  into 
polishing  and  defending  formal  decisions 
that  may  no  longer  be  appropriate  in  fast 
changing  situations. 

Both  of  these  concepts — intuitive 
decisionmaking  and  sense-making — 
can  be  combined  into  a  framework 
for  categorizing  the  residual  thought 
processes  of  intelligence  analysts, 
which,  broadly  following  Weick, 
can  be  called  intelligence  sense¬ 
making.  This  process  involves  the 
application  of  expertise,  imagination, 
and  conversation — and  the  benefit  of 
intuition — within  intelligence  analytic 
organizations  to  identify  changes  in 
existing  patterns  or  the  emergence 
of  new  patterns,  without  systematic, 
consideration  of  alternative  hypotheses. 

Compared  to  analysis,  intelligence 
sense-making  is  continuous  rather  than 
discrete,  informal  rather  than  formal, 
and  focused  more  on  unbounded  rather 


than  on  bounded  issues.  Intelligence 
sense-making  loosely  describes  the 
more  thoughtful  aspects  of  “current 
intelligence,”  in  which  newsworthy 
developments  are  evaluated  in  the 
context  of  past  patterns  usually  without 
the  benefit  of  much  formal  analysis.37 
The  concept  of  “connecting  the  dots” 
also  bears  more  of  an  affinity  to  sense¬ 
making  than  it  does  to  formal  analysis 
in  that  it  involves  holistic  pattern 
recognition,  rather  than  the  testing  of 
alternative  hypotheses. 

Intelligence  sense-making  offers  a  way 
to  comprehend  “complexities,”  whose 
combination  of  speed,  uncertainty,  and 
interactivity  defy  traditional  analytic 
approaches.  One  workshop  presenter, 
a  noted  expert  on  intelligence,  made 
this  point  in  observing  that,  in  contrast 
to  traditional  Soviet-threat  type  issues, 
the  transnational  arena  (terrorism  in 
particular)  benefits  far  more  from  lots 
of  “pairs  of  eyes”  looking  at  data  for 
emerging  signs  of  threat  than  from 
detailed  analyses  of  narrowly  drawn 
issues.  While  lots  of  eyes  would,  of 
course,  be  beneficial  in  addressing 
traditional  issues,  they  arguably  are 
crucial  to  success  in  transnational 
issues,  where  having  multiple  intuitions 
wrestling  with  enormous  information 
flows  should  significantly  improve 
the  odds  in  favor  of  identifying  threat 
patterns. 

“Sense-Making  ”  in  High-Reliability 
Organizations 

Like  traditional  analysis,  intelligence 
sense-making  suffers  from  the  effects  of 
mental  biases,  and  may  be  even  more 
vulnerable  to  those  biases  to  the  extent 
that  it  applies  highly  developed  “mental 
models”  to  incoming  information.  The 
process  of  challenging  assumptions 


The  current  intelligence  process,  as  traditionally  organized,  is  less  collective  and  more  formal  than  the  sense¬ 
making  process  described  by  Weick.  It  is  largely  up  to  individual  analysts  to  make  sense  of  changes  on  their 
“accounts.”  Analysts  do  participate  in  conversations  with  colleagues  and  supervisors  about  their  subjects,  but 
also  engage  in  formal,  sometimes  combative  “coordination”  of  their  written  products  with  counterparts  in  other 
offices  and  agencies. 
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in  this  domain — ’’alternative  sense¬ 
making” — will  be  different,  however,  from 
that  used  in  the  more  familiar  analytic 
domain  where  alternative  analysis 
techniques  strive  to  produce  discrete 
products  using  formal  tools  to  explore  a 
limited  range  of  outcomes. 

A  useful  starting  point  in  seeking  to 
identify  an  “alternative  analysis”  approach 
for  complex  environments  is  Weick’s  work 
with  Kathleen  Sutcliffe.38  The  authors 
examined  a  number  of  organizations, 
such  as  nuclear  power  plants  and 
aircraft  carriers,  which  face  challenges 
in  addressing  uncertainty  akin  to  those 
faced  by  intelligence  organizations. 

Those  organizations  confront  on  a  daily 
basis  continually  shifting  conditions  that 
can  give  rise  to  unexpected  outcomes, 
with  the  potential  for  catastrophic 
consequences.  An  aircraft  carrier  deck, 
they  note,  is  the  “most  dangerous  four 
and  one-half  acres  in  the  world”  with  a 
volatile  mixture  of  jet  fuel,  weaponry, 
“controlled  crash”  landings,  shifting  sea 
and  weather  conditions,  noise,  and  young 
and  often  inexperienced  personnel. 
Nonetheless,  carriers  experience 
exceptionally  low  rates  of  error  and 
accident  in  comparison  to  other  fields 
that  also  continually  confront  uncertainty, 
such  as  medicine.  The  same  can  be  said 
of  nuclear  power  plants  where,  according 
to  another  writer,  “there  is  no  ‘regular’ 
functioning  of  the  plant — the  status  of 
the  plant  is  continually  changing  and 
therefore  . . .  anomalies  are  very  difficult 
to  track.”39 

In  studying  these  types  of  “high-  reliability 
organizations”  (HROs),  Weick  and 
Sutcliffe  have  identified  lessons  that  can 


be  applied  generally  to  addressing 
uncertainty.  The  unifying  trait  of  HROs 
is  that  they  exhibit  the  quality  of 
“mindfulness,”  defined  as: 

“ . . .  the  combination  of  ongoing  scrutiny 
of  existing  expectations,  continuous 
refinement  and  differentiation 
of  expectations  based  on  new 
experiences,  willingness  and  capability 
to  invent  new  expectations  that  make 
sense  of  unprecedented  events,  . . . 
and  identification  of  new  dimensions 
of  context  that  improve  foresight  and 
current  functioning.”40 

The  critical  words  in  the  above  definition 
are  “ongoing”  and  “continuous.” 
Mindfulness  is  the  result  of  a  never- 
ending  effort  to  challenge  expectations 
and  to  consider  alternative  possibilities. 
Such  eternal  vigilance  helps  eliminate 
“blind  spots”  that  result  in  organizations 
missing  early  warning  signs  of 
unexpected  and  unwanted  change.  A 
mindful  orientation  “redirects  attention 
from  the  expected  to  the  irrelevant,  from 
the  confirming  to  the  disconfirming, 

. . .  from  the  more  certain  to  the  less 
certain,  . . .  and  from  the  consensual  to 
the  contested.” 

Weick  and  Sutcliffe  identify  several 
attributes  of  organizational  culture  that 
contribute  to  mindfulness.  Two  that 
appear  to  be  critical  to  anticipating 
uncertainty,  as  opposed  to  reacting 
to  it,  are  most  relevant  to  intelligence 
organizations: 

•  A  preoccupation  with  failure, 
both  past  and  potential. 

•  A  “refusal  to  simplify.” 


See  Managing  the  Unexpected  Assuring  High  Performance  in  and  Age  of  Uncertainty,  (San  Francisco: 
Jossey-Bass,  2001). 

39  Klein,  pp.  132-134. 

This  and  the  following  quotes  from  Weick  and  Sutcliffe  are  found  on  pages  25-62. 
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Underpinning 
these  attitudes 
is  a  ‘learning 
culture’ in  which 
it  is  safe  and 
even  valued ..  .to 
admit  error  and 
raise  doubts. 


The  concept 
of  mindfulness 
. . .  provides 
a  construct 
for  designing 
processes 
to  improve 
understanding 
and  warning 
for  complex, 
transnational 
issues. 


High-reliability  organizations  continually 
seek  information  on  and  then  examine 
their  own  errors,  with  an  eye  to 
improving  operations.  “They  tend  to  view 
any  failure,  no  matter  how  small,  as  a 
window  on  the  system  as  a  whole.”  This 
preoccupation  also  extends  to  thinking 
about  future  downsides:  HROs  tend  to 
be  “skeptical,  wary,  suspicious  of  quiet 
periods,”  because  they  know  that  “not  all 
failure  modes  have  . . .  been  experienced 
or  exhaustively  deduced.”  Underpinning 
these  attitudes  is  a  “learning  culture” 
in  which  it  is  safe  and  even  valued  for 
members  of  the  organization  to  admit 
error  and  raise  doubts. 

The  refusal  of  HROs  to  simplify  involves 
a  refusal  to  take  things  for  granted  or 
to  rely  on  standard  interpretations. 
Personnel  in  HROs  are  relentless  in 
their  efforts  to  try  to  understand  the 
complexities  of  the  situations  they  face 
rather  than  lumping  problems  into  broad 
categories  such  as  “make  or  buy,  friend 
or  enemy,  profit  or  loss."  As  an  example, 
mechanics  in  nuclear  power  plants 
almost  never  rely  on  the  simplifications 
of  blueprints  when  they  have  to 
intervene  in  the  system,  but  rather  they 
personally  “walk  the  system"  to  assure 
that  no  subsequent  changes  have  been 
made  that  could  affect  the  outcome. 
Particularly  relevant  to  the  intelligence 
environment,  different  departments 
constantly  interact  when  confronted  with 
a  problem,  generating  hypotheses  about 
“what  is  going  on,  what  can  be  done, 
and  what  the  long-term,  system-wide 
consequences  of  the  proposed  action 
might  be.”  Such  interdepartmental 
interaction  brings  different  perspectives 
to  the  table,  building  a  progressively 
more  complex  vision  of  the  problem  at 
hand. 

The  sense  of  workshop  participants  and 
the  project  leaders  was  that,  compared 
to  “high  reliability  organizations,”  the 
Intelligence  Community  generally: 


•  Does  not  undertake  the  continuous, 
relentless,  critical  self-examination 
that  is  commonplace  in  HROs,  as  is 
evidenced  by,  among  other  things, 
the  generally  spotty  application  of 
alternative  analysis  techniques; 

•  Has  neither  regularly  explored  its 
own  failings  with  an  eye  toward 
self-improvement  nor  effectively 
implemented  and  internalized 
findings  when  studies  of  failure 
have  been  done. 

•  Contains  analytic  production 
systems  that  promote  simplification 
by  placing  a  premium  on  a  clear 
“bottom  line”  in  intelligence 
products,  with  much  of  the 
complexity  either  eliminated 

or  shunted  to  the  rear,  and  by 
emphasizing  a  consistent  line  on 
a  particular  issue  in  order  to  avoid 
“confusing”  consumers 

•  Has  relatively  less  interaction 
among  analysts  very  early  in  the 
process,  than  is  the  case  in  HROs, 
with  most  of  the  interaction  taking 
the  form  of  formal  coordination 
after  basic  assessments  have  been 
developed. 

Applying  Alternative  “Sense-Making” 
to  Transnational  Issues 

The  concept  of  mindfulness,  as 
practiced  by  HROs,  provides  a  construct 
for  designing  processes  to  improve 
understanding  and  warning  for  complex, 
transnational  issues.  For  threats  that  can 
suddenly  emerge  at  any  time,  anywhere, 
and  in  a  variety  of  forms,  analysts 
need  to  think  more  in  terms  of  a  broad 
mental  readiness  to  perceive  early 
warning  signs  of  threat  than  in  terms 
of  challenging  specific  assumptions  or 
identifying  specific  alternative  outcomes. 
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The  following  principles  can  guide  the 
development  of  an  alternative  process 
for  intelligence  sense-making.  Such  a 
process  must  be: 

•  Continual 

•  Creative 

•  Collaborative 

•  Counterintuitive 

•  Consumer-friendly 

Continual:  The  conventional  model  for 
employing  alternative  analysis — identify 
an  issue  too  important  to  “afford 
getting  it  wrong”  and  then  challenge 
assumptions  and  identify  alternative 
outcomes — is  not  really  suitable  for 
ongoing  complexities.  There  are  too 
many  outcomes  to  be  considered,  too 
much  potential  for  sudden  change,  and 
too  many  contingent  interactions  for 
any  “one-off”  effort  to  be  particularly 
useful.  Moreover,  cognitive  research 
suggests  that  such  efforts  may  not 
make  a  sufficient  imprint  on  thinking 
to  affect  ongoing  analysis.  This  is 
because  information  that  is  inconsistent 
with  expectations  is  less  likely  to  be 
remembered  than  information  that  is 
consistent.  Since  alternative  thinking 
goes  against  the  grain  of  established 
thought,  its  ability  to  have  a  sustained 
impact  on  understanding,  individual  or 
collective,  is  always  open  to  question, 
all  the  more  so  if,  as  is  often  the  case, 
analytic  cadres  turn  over  fairly  rapidly.41 

A  continuous,  sustained  program  of 
small  to  medium-sized  efforts,  however, 
would  regularly  explore  different  possible 
outcomes  and  debate  assumptions,  all 
linked  to  incoming  information  about  the 
issue  under  consideration.  This  probably 


is  best  thought  of  as  an  ongoing 
conversation  (both  face-to-face  and 
electronic)  among  interested  parties, 
structured  to  encourage  divergent 
thinking.  Larger  efforts,  such  as  multiple 
scenario  workshops  or  multiplayer 
games,  would  aim  at  feeding  results 
into  the  ongoing  dialogue,  not  simply 
publishing  and  moving  on.  Information 
technology  to  capture  and  automatically 
recall  both  previous  judgments  and 
alternatives,  perhaps  cued  by  keywords, 
would  be  essential  to  supplement 
human  memory  and  to  further  stimulate 
debate. 

Creative:  Traditional  alternative 
analysis  is  a  fairly  formal  process 
with  some  elements  of  creativity  but 
with  a  strong  emphasis  on  logical 
argument  to  come  to  clear  conclusions. 
Alternative  “sense-making”  for  complex 
issues  would,  by  contrast,  be  more 
freewheeling  and  creative.  In  part,  firm 
conclusions  are  not  desirable  given  the 
higher  levels  of  uncertainty  inherent  in 
these  issues.  In  addition,  the  objective 
is  to  stimulate  pattern  recognition — to 
connect  the  dots — a  creative  process 
in  itself.  Cognitive  research  suggests 
that  judgments  about  the  likelihood  of 
events  often  reflect  the  “availability”  and 
vividness  of  memories  about  similar 
types  of  events.42  To  the  extent  that 
alternative  sense-making  can  help  to 
suspend  premature  judgment  and  make 
an  array  of  possibilities  come  alive,  it 
may  stimulate  exploration  of  alternative 
“dot”  arrangements. 

Increasing  creativity  within  the 
intelligence  field  to  enhance  intuitive 
judgment  can  be  accomplished  in 
a  number  of  ways.  Press  reports 
indicate  that  the  CIA  has  worked, 
for  example,  with  the  film  industry  to 
create  audio-visual  games  to  help 


:  See  Kunda,  pp.  161-210. 
42  Kunda,  pp.  89-101. 
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Ideas  most 
often  ‘pop  out’ 
of  the  slow 
moving,  largely 
unconscious, 
contemplative 
mode  of 
thought  . . . 

. .  .a  state 
of  mental 
relaxation  is 
also  conducive 
to  the  playful 
workings  of  the 
contemplative 
mode . . . 


analysts  “think  like  terrorists.”  Such 
major  efforts  would  have  a  more  lasting 
impact  if  they  were  supplemented  by 
smaller,  more  frequent  efforts  geared 
toward  cultivating  creativity  at  both 
the  individual  and  organizational 
levels.  Systematic  use  of  the  wide 
array  of  divergent  thinking  techniques 
as  a  precursor  to  the  preparation  of 
intelligence  products  would  help  to  put 
a  wide  range  of  possibilities  on  the  table 
before  provisional  intellectual  closure 
was  reached.  Johnston’s  proposal  to 
complement  subject  matter  specialists 
with  analytic  methodologists  might 
be  structured  to  ensure  that  these 
professionals  possess  or  acquire 
expertise  in  the  application  of  such 
divergent  thinking  techniques.43 

A  conscious  mixing  of  mental  biases 
would  also  help  to  assure  that  different 
types  of  “eyes”  are  searching  for 
threatening  patterns.  The  specific 
suggestion  raised  in  workshop 
discussions  involved  the  practice — 
known  as  “barbelling” — used  by  some 
Wall  Street  firms,  which  involves  pairing 
young  financial  professionals  with  those 
over  50  to  marry  adventurousness  with 
caution  born  of  experience.  Firms  with 
such  an  age  distribution,  according  to 
a  noted  presenter  from  the  financial 
community,  tend  to  perform  better  than 
those  whose  professionals  cluster  in 
the  thirty  to  fifty  age  range.  Recruiting 
and  assigning  of  analysts  could  also 
be  designed  to  produce  teams  that 
consciously  mix  biases,  such  as 
perceptual  and  judgmental  thinking 
styles  (as  identified  in  Meyers-Briggs- 
type  tests). 


Providing  analysts  with  much  greater 
time  and  freedom  to  think  about 
problems  than  is  normally  allowed  by 
hectic  intelligence  production  schedule 
could  also  foster  creativity,  as  is 
suggested  by  Claxton’s  findings.  Ideas 
most  often  “pop  out”  of  the  slow  moving, 
largely  unconscious,  contemplative 
mode  of  thought  rather  than  from 
the  more  conscious,  purposeful,  and 
analytic  mode.  Moreover,  a  state  of 
mental  relaxation  is  also  conducive  to 
the  playful  workings  of  the  contemplative 
mode,  something  hard  to  achieve  in 
the  midst  of  a  grueling  production 
schedule.  Other  options  include  enabling 
analysts  to  pursue  independent  projects, 
contacts,  and  conversations  seemingly 
peripheral  to  the  main  tasks  at  hand,  as 
well  as  occasional  working  retreats — for 
physical  setting  is  related  to  creativity  as 
well.44 

Collaborative:  Although  many  forms 
of  traditional  alternative  analysis  are 
collaborative,  an  individual  analyst 
can  do  others,  like  devil’s  advocacy 
or  “what-if”  analysis.  Such  individual 
efforts  are  possible  because  the  aim 
is  to  develop  or  to  challenge  logical 
arguments,  something  that  the  individual 
mind  does  well.  However,  examining 
alternatives  on  a  sustained  basis  for 
transnational  issues  (and  for  comparable 
traditional  issues)  involves  a  team 
effort  embracing  a  variety  of  disciplines 
and  regional  specialties — far  beyond 
the  capabilities  of  a  single  analyst. 
Moreover,  introspection  is  extremely 
difficult  for  most  individuals,  as  a  leading 
cognitive  psychologist  noted  at  one 
workshop,  and  is  amply  documented 
in  psychological  literature.  To  promote 
sustained  awareness  of  possible  flaws 


For  a  discussion  of  creativity  in  an  organizational  context  see  John  Kao,  Jamming-.  Art  and  Discipline  of 
Corporate  Creativity,  (New  York:  Harper  Business,  1997). 

Claxton,  pp.  201-226.  Also  see  Judith  H.  Heerwagen,  “Creativity”  at  http://www.science.doe.aov  (last  visited 
July  13,  2004). 
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in  one’s  thinking  requires  a  continuous 
dialogue  among  individuals  with  different 
perspectives. 

One  collaborative  technique  discussed 
at  length  at  the  workshops  was  “out  loud 
sense-making” — a  structured  process 
for  orally  reviewing  assumptions  and 
alternatives.  Such  a  process  is  used 
by  surgical  teams  at  a  New  England 
hospital,  in  response  to  recurrent  errors 
by  anesthesiologists  in  diagnosing 
respiratory  reactions — often  fatal — to 
anesthesia.  Now,  when  confronted  with 
breathing  problems,  the  team — including 
nonphysicians — systematically  reviews 
possible  causes  out  loud  before 
arriving  at  a  conclusion,  dramatically 
reducing  the  error  rate.  Such  processes, 
enhanced  by  skilled  custom  design  and 
facilitation,  could  be  regularly  employed 
in  the  intelligence  arena  whenever 
anomalous  data  emerged. 

Another  collaborative  technique 
identified  at  the  workshops  involves  the 
greater  use  of  Web-logs  (or  “blogs”)  as 
a  platform  for  intelligence  production.  US 
embassies  already  use  this  art  form — 
which  allows  for  continuous  posting  of 
information — as  their  primary  means  for 
conveying  information.  Unlike  published 
papers,  intelligence  Web-logs  would 
be  a  form  of  “unfinished”  production  in 
which  both  intuitions  and  more  formal 
arguments  could  be  posted,  and  then 
challenged  by  those  with  alternative 
opinions.  Indeed,  Web-logs  could  be  the 
forum  for  a  facilitated  virtual  dialogue — 
the  electronic  equivalent  of  out  loud 
sense-making. 

Counterintuitive:  Intelligence  analysts 
need  to  focus  on  patterns  that  are 
different,  even  contradictory  to  those 
they  expect,  to  lessen  the  degree  to 
which  their  mental  models  inhibit  their 
ability  to  perceive  new  information. 


And  they  must  do  so  fairly  regularly 
to  promote  continuous  awareness  of 
possibilities.  This  is  not  easy  given  the 
natural  human  tendency — especially 
under  time  pressure — to  revert  to 
established  ways  of  viewing  the  world. 

One  possible  approach  is  to 
institutionalize  brief,  informal  exercises 
in  which  analysts  regularly  focus  on  how 
they  could  be  wrong,  along  the  lines  of 
what  Klein  calls  a  “premortem.”45  In 
this  group  exercise,  individuals  imagine 
fiascoes  relating  to  their  areas  of 
responsibility — in  the  case  of  intelligence 
analysis,  a  warning  failure — and 
brainstorm  about  how  they  might  come 
about  and  how  they  could  be  headed 
off.  The  very  pointed  emphasis  on  the 
negative,  Klein  argues,  helps  to  shake  up 
complacency  that  arises  from  common 
overconfidence  in  judgments.  Such  an 
exercise  would  be  a  functional  equivalent 
of  the  “low-probability/high-impact” 
exercise  of  traditional  alternative  analysis, 
with  the  difference  that  its  informal  nature 
lends  it  to  more  frequent  use. 

Workshop  participants  also 
discussed  changing  the  culture  within 
intelligence  organizations  to  one  that 
is  more  conducive  to  self-questioning. 
Assessment  of  analytic  performance 
often  is  done  by  outside  bodies  with 
negative  findings  exploited  by  media 
and  external  critics,  thus  prompting  a 
defensive  stance  among  intelligence 
professionals.  There  has  been  only 
modest  emphasis  internally  on 
looking  at  failures — and  even  less  on 
examining  successes — with  an  eye  to 
drawing  lessons  for  self-improvement, 
notwithstanding  Sherman  Kent’s 
exhortation  for  such  introspection  more 
than  40  years  ago. 


Cited  above,  pp.  88-91 . 
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The  systematic  and  non-punitive 
approach  of  the  nuclear  power  industry 
to  collecting  information  on,  and  learning 
from,  errors — associated  with  its  low 
failure  rate — provides  one  approach 
worth  considering.  Another,  perhaps 
more  relevant,  example  is  provided  by 
the  US  Army’s  Center  for  Army  Lessons 
Learned  (CALL),  which  continuously 
looks  at  operational  problems  in  order 
to  draw  implications  for  improving 
performance.  A  dedicated  effort  to 
continually  take  a  friendly  look  at  getting 
it  both  right  and  wrong  through  periodic 
“After  Analysis  Reviews” — modeled  on 
the  Army’s  After  Action  Review — might 
help  to  create  a  more  introspective 
environment.46  Allowing  analysts  the 
time  for  such  exercises — or  mandating 
them — would  powerfully  demonstrate 
organizational  commitment  to  learning. 

Consumer-friendly:  One  of  the 

greatest  challenges  of  any  “alternative” 
effort  is  to  effectively  communicate 
the  message  to  those  who  occupy 
decision-making  roles.  Decision¬ 
makers  are  buried  by  both  information 
and  tasks.  Motivating  them  to  spend 
time  reading  sophisticated  analysis  in 
general,  let  alone  analysis  that  queries 
existing  analytic  lines,  is  a  considerable 
challenge.  Several  presenters  at  the 
workshops  who  had  been  senior  officials 
in  the  terrorism  area  stressed  the  extent 
to  which  information  overload  had  grown 
in  the  post-September  11  environment. 
Moreover,  in  the  transnational  domain, 
many  potential  key  consumers  are  in 
middle  and  lower  operational  levels,  or 
outside  the  government,  and  thus  have 
even  fewer  contexts  for  understanding 
intelligence  information.  And,  of  course, 
in  order  to  carry  out  “alternative  sense¬ 
making,”  alternative  points  of  view  would 
need  to  be  put  before  this  wide  array 
of  harried  individuals  on  a  fairly  regular 
basis. 


One  way  to  accomplish  this  is  to  rethink 
the  concept  of  the  intelligence  “product.” 
Intelligence  organizations  continue  to 
insist  upon  written  prose  and  formal 
briefing  as  the  “gold  standard”  for 
disseminating  information  even  though 
adults  rarely  retain  more  than  ten  percent 
of  what  they  are  “told”  either  orally  or  in 
written  form.  Instead,  more  experiential, 
interactive  formats,  as  discussed  at  the 
workshops,  might  better  capture  the 
attention  and  imagination  of  intended 
audiences  and  strengthen  retention  of 
insights. 

•  Use  of  Web-logs  would  give 
consumers — particularly  nonsenior 
consumers  without  formal  feedback 
processes — the  opportunity  to  tap  in 
from  time  to  time  on  debates  within 
the  analytic  community  and  to  pose 
questions  themselves. 

•  RapiSims — i.e.  Rapid  Simulation 
using,  increasingly  sophisticated 
spreadsheet-based  programs — 
would  allow  consumers  to 
manipulate  variables  to  generate 
alternative  outcomes.  Decision¬ 
makers  could  quickly  and  easily 
explore  a  range  of  possibilities  in  a 
way  that  is  more  likely  to  be  retained 
than  if  presented  in  a  long  and  dry 
formal  tome. 

•  Half  day  “gaming”  sessions — 
intentionally  kept  brief  to  allow  even 
the  most  harried  to  participate  on 
occasion — could  help  decision¬ 
makers,  at  a  minimum,  experience 
the  uncertainties  surrounding  an 
issue. 

Another  avenue  would  be  to  try  to 
strengthen  personal  relationships, 
such  as  through  increased  face-to-face 
contacts,  between  individual  analysts  and 
consumers  in  order  to  facilitate  informal 
exchanges  on  alternative  outcomes. 


46  For  a  discussion  of  the  Army’s  after  action  process,  see  Nancy  M.  Dixon,  Common  Knowledge:  How 
Companies  Thrive  By  Showing  What  They  Know ,  (Boston,  Harvard  Business  School  Press,  2000),  pp.37-38. 
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One  presenter,  a  specialist  in  corporate 
innovation,  argued  that  the  real  product 
of  the  analytic  organization  is  the 
“analyst  rather  than  the  analysis,”  just 
as  professionals  in  other  knowledge 
industries — management  consulting, 
law,  etc. — are  the  key  selling  points  for 
their  organizations  rather  than  specific 
products.47  A  professional  known  by 
and  enjoying  the  trust  of  consumers 
is  in  a  far  better  position  to  influence 
their  thinking  than  is  any  specific  report 
or  briefing.  Such  a  professional,  if 


inculcated  in  alternative  sense-making 
values  and  processes,  also  would 
be  in  a  position  to  periodically  draw 
consumers’  attention  to  the  results  of 
internal  exercises  on  assumptions  and 
outcomes. 

Some  of  the  practical  ideas  stemming 
from  workshop  discussions  are 
summarized  in  Table  2. 


Table  2:  Key  Practical  Ideas 


Idea 

Implementation  and  purpose 

Develop  information  technology  to  store 
and  automatically  recover  hypotheses, 
ideas. 

Aid  analysts’  memory  and  creative  thinking, 
and  promote  collaboration. 

Employ  analytic  methodologists  with 
training  in  creativity  and  facilitation. 

Design  and  facilitate  divergent  thinking 
exercises  and  structured  dialogues  aimed 
at  surfacing  alternative  views. 

Consciously  mix  biases  in  teams  (e.g. 
“barbelling”). 

Increase  likelihood  of  alternative 
interpretations  of  evidence. 

Introduce  “out  loud  sense-making” 
processes,  both  in  person  and  virtual. 

Structured  dialogues  to  consider  all 
possibilities. 

Use  Web-logs  as  a  production  vehicle. 

Common,  continuous  platform  for  carrying 
out  a  “virtual  dialogue”  on  alternatives. 

Regularly  do  after-action  reports. 

Look  at  failures  and  successes  with  an  eye 
to  drawing  constructive  lessons. 

Provide  opportunities  for  experiential 
learning  by  intelligence  consumers. 

Brief  simulations/games  to  help  consumers 
comprehend  range  of  uncertainty. 

Promote  analyst  reflection  and 
introspection. 

Allow  time  off-line  for  premortems  and  after¬ 
action  exercises. 

47  Former  CIA  Deputy  Director  for  Intelligence  Winston  Wiley  often  said  that  the  Directorate  of  Intelligence 
produces  two  products:  intelligence  analysis  and  intelligence  analysts. 
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Conclusion 

Understanding  the  complexities  facing 
analysts  today  requires  an  alternative 
analysis  approach  that  is  more  an 
ongoing  organizational  process  aimed 
at  promoting  continuous  “mindfulness” 
than  it  is  a  set  of  tools  that  analysts  are 
encouraged  to  employ  occasionally.  The 
latter  approach  lays  the  responsibility 
for  alternative  analysis  too  heavily  on 
the  shoulders  of  individual  analysts  who 
may  not  have  the  incentives  or  authority 
to  question  their  own  or  their  colleagues’ 
lines  of  thinking.  And  in  the  case  of 
highly  uncertain,  rapidly  changing, 
boundary-crossing  threats  such  as 
those  in  the  transnational  arena,  such 
questioning  can  really  take  place  only 
as  part  of  an  ongoing,  organizationally- 
supported  collaborative  effort,  given  the 
wide  range  of  disciplines  and  institutions 
that  contribute  to  shaping  the  ultimate 
analytic  product. 

With  the  global  landscape  likely  to 
feature  a  growing  array  of  hard-to-track 
threats  in  the  years  ahead — as  a  result 
of  the  increasing  availability  of  technical 
know-how  and  soaring  interconnectivity, 
among  other  factors — the  need  for 
a  process  to  produce  sustained 
mindfulness  will  only  grow.  Even  in  the 
case  of  more  traditional  intelligence 
issues,  developing  and  enhancing 
organizational  processes  designed 
to  promote  alternative  analysis  might 
encourage  more  systematic  use  of 
these  tools. 


have  been  wrong.  Production  schedules 
and  staffing  requirements  need  to 
be  adjusted  to  allow  time  for  analytic 
reflection.  Throughout  the  organization 
there  must  be  a  “culture,”  one  inculcated 
by  exhortation,  training,  and  example, 
that  values  continuous,  collective 
introspection. 

Rethinking  the  concept  of  alternative 
analysis  may  even  mean  rethinking 
how  the  intelligence  field  defines  itself. 
For  instance,  the  CIA’s  Directorate 
of  Intelligence  has  self-consciously 
modeled  itself  after  the  academic  realm 
and  to  a  lesser  extent  the  press,  where 
questions  of  accuracy — getting  the 
data  and  facts  right — are  of  paramount 
importance.  But  should  it  not  think  of 
itself  as  more  equivalent  to  an  aircraft 
carrier  or  nuclear  power  plant,  where 
questions  of  failure  drive  every  aspect  of 
operations? 

Would  increased  mindfulness  reduce 
the  risk  of  surprise?  Alternative  “sense¬ 
making”  is  but  one  component  of 
the  warning  system,  which  includes 
collection,  information  sharing,  and 
dissemination.  Inadequacies  in  any  of 
these  areas  will  undermine  even  the 
most  mindful  analytic  efforts.  Moreover, 
the  environment  that  intelligence 
seeks  to  comprehend  is,  arguably, 
considerably  more  complicated  than 
that  facing  an  aircraft  carrier,  with 
uncertainties  of  every  type — coupled 
with  active  attempts  to  deceive — 
compounding  the  analytic  challenge. 


Thinking  of  alternative  analysis  as  an 
organizational  process  has  important 
implications  for  how  the  intelligence 
analytic  organization  is  managed.  It 
means  that  senior  intelligence  officials 
must  make  challenging  assumptions 
and  considering  alternative  possibilities 
a  high,  indeed  critical,  priority.  It  calls  for 
reward  systems  that  encourage  analysts 
to  think  about  how  they  could  be  or 


Increased  mindfulness  is  not  a 
panacea.  Nonetheless,  a  look  at 
some  past  surprises  in  “complex” 
situations  suggests  that  even  modest 
improvements  from  incorporating 
mindfulness  could  make  a  significant 
difference  in  preparedness. 

•  In  the  case  of  Pearl  Harbor,  as 
Cohen  and  Gooch  have  pointed 
out,  small  enhancements  in  military 
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readiness,  like  deploying  barrage 
balloons  and  torpedo  nets  in  the 
harbor,  could  have  resulted  in  a 
significant  reduction  of  losses.48 
Would  these  steps  have  been  more 
likely  taken  had  the  findings  of  the 
alternative  analysis-type  study  of  a 
surprise  carrier  attack  noted  earlier 
been  integrated  into  an  ongoing 
process  of  reexamination,  instead 
of  being  consigned  to  a  dusty  file 
shelf? 

•  In  the  case  of  the  runup  to  the 
Battle  of  France,  examining  the 
pattern  of  German  reconnaissance 
flights  would  have  highlighted 

a  likely  invasion  route,  perhaps 
resulting  in  redeployment  of  French 
forces  to  their  weakest  sector — the 
Ardennes.  Had  there  been  an 
ongoing  process  of  reexamining 
assumptions  in  light  of  incoming 
information — some  of  which 
suggested  the  Ardennes  possibility 
— might  not  that  critical  analysis 
have  taken  place?  Interestingly, 
the  then  head  of  French  military 
intelligence  wrote  years  later  that — 
evidently  still  not  comprehending 
an  important  contributor  to  the 
intelligence  debacle — "whoever 
works  in  intelligence... must  avoid 
being  led  by  intuition  or  excessive 
imagination”.49 

•  On  September  11th,  about  half 
of  the  hijackers  had  been  flagged 
for  scrutiny  at  the  gate  before 
boarding  the  ill-fated  flights.  Had 
the  concerns  of  the  Phoenix  FBI 
office  about  flight  training  not  only 
been  shared  broadly  within  the 
government  but  also  integrated 
into  a  mindfulness-focused  inter¬ 
agency  process — featuring  out  loud 


sense-making,  Web-log  type  forums, 
computer-generated  references  to 
extant  scenarios  for  crashing  airplanes 
into  prominent  targets — might  at 
least  some  of  the  detentions  been 
prolonged,  disrupting  the  plan? 

Counterfactual  questions  cannot,  of  course, 
yield  firm  answers,  but  if  they  pique  curiosity 
or  create  unease,  it  is  a  signal  that  further 
exploration  of  the  issue  at  hand  is  warranted. 
This  is  the  case  for  alternative  analysis  of 
transnational  threats. 


48Cohen  and  Gooch,  pp.50-51 
49May,  p.365 
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